This is Info file database.info, produced by Makeinfo-1.55 from the input file database.texi.  File: database.info, Node: Manipulating database fields, Prev: Creating a new database, Up: Designing a database Manipulating database fields ============================ EDB will have a graphical method to manipulate databses by adding, removing, and rearrange fields, among other other manipulations; a rudimentary version of this functionality exists in the `db-convert.el' file but lacks a good user interface or documentation. Here is a way to perform those actions by directly manipulating the database file. Make sure the database is stored in a regular layout (the rest of this section assumes that the standard tab-separated text layout is used; *note Regular file layout::.). If the database file is in EDB's internal file layout, convert it to a regular layout by reading in the database, setting database slot `internal-file-layout-p' to `nil' (for instance, via `C-u 0 M-x db-toggle-internal-file-layout RET'), and saving the database. (*Note Internal file layout::.) In the tab-separated database format, you can add new fields by adding new tabs in the right places in each record; simply edit the database file to add the next fields as desired. Similarly, you can delete or rearrange fields. You may wish to use keyboard macros, or write an Emacs Lisp function, to help youwith this chore. When you perform this edit, be sure that auto-fill-mode is turned off, lest spurious newlines be added to the file. You will also need to modify any parts of your database that depend on the number or order of fields; for instance, a call to `database-set-fieldnames-to-list' will need to be changed, and the format file(s) should be edited if you wish to be able to view or edit the contents of the new fields.  File: database.info, Node: Record field types, Next: Database file layout, Prev: Designing a database, Up: Top Record field types ****************** Each of a database's records consists of similarly-typed fields: the fifth field might always contain an invoice number, for instance, but the invoice number would vary from record to record. This chapter describes how to specify and use different field types. The most important information about a record field is * display representation onscreen and in reports * EDB's internal representation * storage representation in data files * how to convert among these representations * how to sort items of that type This information is separated into a display specification and a record field specification. The display specification determines how a field's contents are displayed and parsed onscreen (say, in a data display buffer). The record field specification controls everything else about the record field; its information does not depend on the onscreen (or in-report) visual appearance of the field. The database designer specifies a displaytype for each display specification in the format file (that is, for each location in the data display buffer that will contain a representation of some record field). The database designer specifies a record field type for each field in a database record, whether or not the field is ever displayed to the user. (A particular field's contents may appear zero, one, or more times in a data display buffer; one displayspec structure is created for each occurrence.) Displaytypes and record field types are distinct; they supply complementary information. There is not even a one-to-one relation between them. A particular record field type may be displayed in any of a number of ways by using different displaytypes--dates are such an example. On the other hand, record field types which are interpreted, sorted, and stored on disk differently, but which have the same internal representation--say, as a string--could all be displayed and edited using the same displaytype. This chapter discusses record field types, record field specifications, and the recordfieldspec structure, the internal structure which holds the information specified by the former two items. For more information about displaytypes, display specifications, and the displayspec structure, *note How information is displayed::.. * Menu: * Specifying a record field type:: Specifying a record field type * Predefined record field types:: Predefined record field types * The recordfieldspec structure:: The recordfieldspec structure  File: database.info, Node: Specifying a record field type, Next: Predefined record field types, Up: Record field types Specifying a record field type ============================== The function `database-set-fieldnames-to-list' (*note The database structure::.) is used to specify the types (and names) of record fields. A record field type gives information about one field of the database's records: the type of the contents, what sorting function to use, how to write it to disk and read it back, constraints on its value, etc. The database designer must provide one for each record field. Most of the time one of the predefined types (*note Predefined record field types::.) suffices. The remainder of this section describes how to define a new record field type, when that is necessary. The first step in creating a new record field type is to make a recordfieldspec structure (when a predefined record field type is used, EDB looks up an existing recordfieldspec). A recordfieldspec can be created from scratch by calling `make-recordfieldspec', but it is often easier to copy an existing one with `copy-recordfieldspec' (use `recordfieldtype->recordfieldspec' to look up a predefined recordfieldspec; for a list of such, *note Predefined record field types::.) and then modify the copy as appropriate. `recordfieldtype->recordfieldspec' Return the recordfieldspec associated with symbol RECORDFIELDTYPE. Next, set the recordfieldspec's slots to appropriate values by using `recordfieldspec-set-SLOTNAME'; for a list of the slots, *note The recordfieldspec structure::.. Finally, install the record field type by calling `define-recordfieldtype-from-recordfieldspec' `define-recordfieldtype-from-recordfieldspec' Define a recordfieldtype named TYPENAME (a symbol) with the default RECORDFIELDSPEC. DISPLAYSPEC may also be a typename symbol itself. After this call, `recordfieldtype->recordfieldspec' called with argument TYPENAME returns the proper record field specification. Examples of record field type creation can be found in the file `db-types.el', which contains a number of record field type (and displaytype) definitions that can be studied or copied. Record field types should not be confused with display types; a display type is used to specify how a particular value is shown on the screen, but a record field type constrains the information actually contained in the record field.  File: database.info, Node: Predefined record field types, Next: The recordfieldspec structure, Prev: Specifying a record field type, Up: Record field types Predefined record field types ============================= The following record field types are predefined by EDB; their definitions can be found in the file `db-types.el'. The recordfieldspec structure contains a record field type's information; see *Note The recordfieldspec structure::. Users can define record field types in the same way as `db-types.el' does; these record field types are not privileged in any way and are provided only for convenience. For more information about recordfieldspec creation, see *Note Specifying a record field type::. integer Ordinary integers. integer-or-nil Integers or `nil', the empty value; by default, `nil' is treated as larger than any integer, so it comes last in an increasing-order sort. number Ordinary numbers. A number is either an integer or a floating-point number. number-or-nil Numbers or `nil', the empty value; by default, `nil' is treated as larger than any number, so it comes last in an increasing-order sort. boolean This displayspec corresponds to the yes-no displaytype. For the purposes of sorting, `t' is considered less than `nil', so it appears first in a sort in increasing order. string Ordinary strings. one-line-string Strings which may not contain newlines. string-or-nil Either a string or the value `nil', which is converted to the empty string. Sorting treats `nil' identically to the empty string. nil-or-string Identical to the string-or-nil recordfieldspec (except for the name). This exists so that display fields of type nil-or-string can conveniently default to this recordfieldspec. one-line-string-or-nil The obvious combination of the one-line-string and string-or-nil recordfieldspecs. date A date which specifies zero or more of the year, month, and day. By default the date is sorted by year, then month, then day; an unspecified component is larger than any specified component ("March 14, 1967" would appear before "January 1" if dates were sorted in increasing order). Dates are read from database files using the function value of `storage-string->date', which is set by default to `date-stored->actual', which can parse nearly any string representation of a date and returns a date if it is passed one. Dates are written using the function value of `storage-string->date', which defaults to `format-date-full', which produces a string of the form "March 14, 1967". If the speed of reading and writing database files is very important to you, consider using `fset' to set `date->storage-string' and `storage-string->date' to more efficient functions, such as `date->storage-string-mmddyyyy' and `storage-string-mmddyyyy->date', or `date->storage-string-lisp' and `storage-string-lisp->date'. date-or-nil A date, or nil. date-efficient-storage When the dates in a database file are known to have a particular format, using `parse-date-string' is unnecessarily inefficient. The date-efficient-storage recordfieldspec specifies that `storage-string->date', which can efficently read dates written by `simple-format-date', be used instead. The time savings is noticable on large databases. time A time. If you find any of these typenames cumbersome, you can create your own aliases for them using `define-displaytype-from-displayspec', `define-recordfieldtype-from-recordfieldspec', or `define-type-alias'. `define-type-alias' Make symbol ALIAS refer to the same displaytype and recordfieldtype as TYPENAME.  File: database.info, Node: The recordfieldspec structure, Prev: Predefined record field types, Up: Record field types The recordfieldspec structure ============================= The recordfieldspec structure contains information regarding the content of a record field, but nothing concerning how it is displayed on the screen or read from user input. The slots of a recordfieldspec are listed below; a slot may be accessed by using the macro `recordfieldspec-'SLOTNAME and set using the macro `recordfieldspec-set-'SLOTNAME, whose second argument is the value to be stored in the slot. Most of the slots may be left `nil', and reasonable default actions will occur. `type' A symbol such as `string' or `integer', the type of the data described by this recordfieldspec. When no displaytype is explicitly specified in a display specification, then a displaytype with the same name as `type' slot is used by default; this is the only use for this slot. `default-value' The default value for fields described by this recordfieldspec; used when creating new records. `common-form-function' A function which, called on the contents of a record field, returns them in canonical form. This can be used for determining non-trivial equality, when two nonidentical values should be considered equivalent. `merge-function' A function which, called on the contents of two record fields, returns a combination of the two. Often it queries the user for help. `order-fn' `sort-fn' The record field's ordering and sorting functions (*note Sorting functions::.). Both ordering and sorting are possible if either slot is filled. If both slots are empty, then a dummy ordering or sorting function is used, so sorting on this field has no effect. Since the function is called and its result examined, this is more expensive than not sorting on the field in the first place. If it does not make sense to sort on a particular field, then it is best to keep that field out of the field priorities used for sorting (which is the `field-priorities' database slot, or is interactively specified through the database sort interface (*note Sorting::.), or is specified as an argument to `database-sort'). Users may set the `order-fn' and `sort-fn' slots directly, but should use the following functions to access them: `recordfieldspec-order-function' Return an order function for records described by RECORDFIELDSPEC. If optional argument REVERSEP is non-`nil', then the order function goes in the opposite order. If the order-fn slot of the appropriate recordfieldspec of database doesn't contain one, one is made up on the fly from the sort-fn slot; `equal' is used to determine whether two records are equal. If the sort-fn slot is also empty, the resulting function always returns 0, indicating equality. `recordfieldspec-sort-function' Return a sort function for records described by RECORDFIELDSPEC. If optional argument REVERSEP is non-`nil', then the sort function goes in the opposite order. If the sort-fn slot of the appropriate recordfieldspec of database doesn't contain one, one is made up on the fly from the order-fn slot. If the order-fn slot is also empty, the resulting function always returns `nil', indicating that it is not the case that the first argument is less than the second. `match-function' A function which takes a pattern and a field value and returns non-`nil' if they match. The function should also be able to take as its first argument a field value rather than a pattern. `help-info' A string which is displayed by `db-field-help' when there is no field-specific help available. Field-specific help is usually preferable to this help, which only describes the type of the field's contents. `actual->stored' A function which converts a field value into its on-disk representation (a string). `stored->actual' A function which recovers a field value from its on-disk representation (a string). If this function returns a string, it should return something reasonable if supplied the empty string as its argument. (That can happen when an empty database is read.) `change-hook' A function called when the value of this field is changed. This is not currently implemented. For more change hooks, *note Display format change hooks::., and *note Recordfieldspec change hooks::.. `constraint-function' A function which the value of this field must satisfy; that is, the function must return non-`nil' on it. The function may reject the value either by returning `nil' or by signalling an error; the latter permits the function to provide an informative message about the problem. Four arguments are supplied to `constraint-function': the field value, the record, the record fieldnumber, and the database. This permits cross-field and cross-record constraints. The record argument may be `nil', in which case the function should return `t' if the value is acceptable for some conceivable record. This occurs, for instance, when values are read in a call to `db-field-query-replace'. The constraint function may interact with the user; for instance, it may give the use the opportunity to override the constraint.  File: database.info, Node: Database file layout, Next: How information is displayed, Prev: Record field types, Up: Top Database file layout ******************** This chapter discusses specifying how a database is read from a file (or saved back to it). Broadly stated, there are three possible file representations for a database: EDB's internal file layout, a regular layout, or a nonregular layout. EDB's internal file layout is designed for fast reading and writing, but is not very human-readable. A regular layout is one in which records (and fields within a record) are separated from one another in a predictable (though not necessarily invariant) way. A nonregular layout is any other kind of layout; the user may specify arbitrary Emacs Lisp code to read and write such files. Support for tagged file layout (a special case of nonregular file layout) is included with EDB. If the database is to be stored in EDB internal file layout, a lot of this information is not needed except when the database is first created. The sections of this chapter each describe a file layout, except the last, which describes in detail the process of reading a database from disk. * Menu: * Internal file layout:: Internal file layout * Regular file layout:: Regular file layout * Tagged file layout:: Tagged file layout * Nonregular file layout:: Nonregular file layout * Reading from disk:: What happens when a database is read in from disk  File: database.info, Node: Internal file layout, Next: Regular file layout, Up: Database file layout Internal file layout ==================== The first line of a database file in EDB's internal file layout looks something like ;; Database file written by EDB; format 0.3 followed by the printed representations of two Emacs Lisp forms, a record (the database structure) and a list of records (the records of the database). Databases stored in this layout can be loaded and saved very quickly (sometimes orders of magnitude faster than databases which EDB must parse when reading), and they never suffer from ambiguities between data and delimiting text, but they are not easy for people to read and understand. A human- or program-readable version of the database can be generated when it is needed, either by creating a report or by saving in some other file layout. This is a good option when all manipulation of a database will be done via EDB. Since this file layout is rather complicated, databases are often created in some other file layout and then converted to this one. To convert from another file layout to EDB's internal file layout, read in the database, set database slot `internal-file-layout-p' to `t' (for instance, via `C-u M-x db-toggle-internal-file-layout RET'), and then write or save the database (via `C-x C-w' or `C-x C-s'). Convert a database from EDB file layout to some other representation is similar, but slot `internal-file-layout-p' is set to `nil' (say, via `C-u 0 M-x db-toggle-internal-file-layout RET'), and certain variables and database values may need to be set (see the documentation for the layout you desire, elsewhere in this chapter). Making a report can also convert a database to a different file layout, with even more flexibility than the techniques described here. A database file in EDB internal file layout is basically the printed representation of the Lisp database structure used by EDB when the database is read in. As such, it contains all the information in the database slots described in *Note The database structure::, except that the `data-display-buffers' and `first-link' slots are set to `nil'; in the file, the records follow the database structure. After a database has been saved in internal file layout, then any forms in the auxiliary or format files that set these slots can be removed if desired; this is not necessary, however.  File: database.info, Node: Regular file layout, Next: Tagged file layout, Prev: Internal file layout, Up: Database file layout Regular file layout =================== EDB can conveniently read and write database files in which records are separated from one another by a record delimiter and, within each record, fields are separated by a field delimiter. When the delimiters are the newline and tab characters, respectively, the result is the standard "tab-separated text" layout, which is often used for transferring information from one program to another. The record and field delimiters need not be single characters; they can be arbitrary strings or can even be specified by a regular expression instead of a particular string. The latter is useful if the exact delimiter is not known ahead of time (for example, if records may be separated by one or more carriage returns). This regular expression mechanism can only be used when reading the database: when writing a database, all the record and field delimiters will be identical. (Exception: you may specify an arbitrary record-writing function (*note Nonregular file layout::.) and arbitrary functions for either reading records or for separating records or fields in regular layout, but should use the simpler reading mechanisms whenever possible, for your own sake.) Regular file layout has two disadvantages. First, it is somewhat slower to read and write than EDB's internal representation. Second, the strings used as delimiters may not appear in the database fields, lest those occurrences be misinterpreted as delimiters rather than as data. EDB provides two solutions to the latter problem: substitution and quoting (*note Resolving ambiguities::.). * Menu: * How to specify regular file layouts:: How to specify regular file layouts * Resolving ambiguities:: Resolving ambiguities * Problems with end-of-file newlines:: Problems with end-of-file newlines  File: database.info, Node: How to specify regular file layouts, Next: Resolving ambiguities, Up: Regular file layout How to specify regular file layouts ----------------------------------- In a database stored in regular file layout, records and fields can be separated by particular strings, by regular expressions, by context-sensitive regular expressions, or by arbitrary functions. The sepinfo structure holds this information for use when reading the database from disk (and writing it back). The sepinfos used when reading a database are stored in its `record-sepinfo' and `field-sepinfo' slots (for more details about the database structure, *note The database structure::.). When reading, if a separation function is specified, it is used; otherwise, if a regular expression is specified, it is used; otherwise, a string must be specified. It is converted into a regular expression, the regular expression slots of the sepinfo are filled in, and reading proceeds as if the user had specified a regular expression. (A user who wishes to have a regular expression recomputed when it is next needed should set it to `nil' when setting the corresponding string value.) When a sepinfo is used for writing, it must specify literal string separators. (The sepinfo may have its separation function or regular expressions set as well, for reading, but those slots are ignored when writing.) * Menu: * The sepinfo structure:: The sepinfo structure * Sepinfo examples:: Examples of setting record and field separators  File: database.info, Node: The sepinfo structure, Next: Sepinfo examples, Up: How to specify regular file layouts The sepinfo structure ..................... The sepinfo structre contains the information required to decide where records or fields start and end (actually, to determine where the record or field separators start and end; "sepinfo" is short for "separator information"). The slots of this structure may be accessed by using the macro `sepinfo-'SLOTNAME. The slots may be set using the macro `sepinfo-set-'SLOTNAME, whose second argument is the value to be stored in the slot. The `pre-first-' slots describe text that precedes the first item of interest. In a record sepinfo they describe the file header, which precedes the first record. In a field sepinfo they describe any information that preceded the first field of every record, after the record separator. The `post-last-' slots are similar, but are used to inform EDB of text following the last information-carrying text. In a record sepinfo, they describe the file trailer, which follows the last record in the file. A field sepinfo's `post-last-' slots tell about information following the last field of a record but preceding the record separator. The `-submatch' integers describe which submatch of a regexp match is the actual separator, as opposed to surrounding text used to help make the match unambiguous. This specification of the submatch permits context-sensitive matching that you might otherwise expect could not be done with regular expressions alone. For instance, suppose a database has records with a variable number of fields separated by newlines, that records are also separated by newlines, and that the first field of each record has some special form different from all other fields (say, it is a number with a decimal part). The following code would permit separation of the records without writing a special function to do so and without including the decimal number in the separating text: (sepinfo-set-sep-regexp (database-record-sepinfo database) "\\(\n\\)[0-9]+\\.[0-9]+") (sepinfo-set-sep-regexp-submatch (database-record-sepinfo database) 1) When you set the slots of the sepinfo, be careful to use a correct value. For instance, if your record separator is a form feed on a line by itself, you probably want to set the `sep-string' slot of the database's `record-sepinfo' to `"\f\n"', or possibly `"\n\f\n"', rather than just `"\f"', lest the newlines be considered to be part of the records rather than part of the separator. The slots of the sepinfo are listed below but are not described in detail; see the preceding description for details of their use. `pre-first-string' Setting the slot to `nil' (or not setting it) is equivalent to setting it to the empty string. `pre-first-regexp' `pre-first-regexp-submatch' `sep-string' `sep-regexp' `sep-regexp-submatch' `sep-function' A function that takes a buffer position, the end of the previous separator (that is, the start of the current record or field), as its argument and returns a pair of two buffer positions bracketing the next separator. That is, the returned values are the end of the current record of field and the beginning of the next one (or nil if there are no more). When the function is called, point is at the beginning of an item and the buffer is narrowed to the list being currently processed. The use of a separation function is useful when the separation criteria cannot be expressed as a combination of regexp expressions. The `pre-first-' and `post-last-' slots are still used even if a function is specified. `post-last-string' Setting the slot to `nil' (or not setting it) is equivalent to setting it to the empty string. `post-last-regexp' `post-last-regexp-submatch'  File: database.info, Node: Sepinfo examples, Prev: The sepinfo structure, Up: How to specify regular file layouts Examples of setting record and field separators ............................................... [[[Put examples here.]]] [[[ For instance, to parse "[Mary, John,Jack, and Jill]" and to write it back out as "[Mary, John, Jack, Jill]", the following specification would suffice: pre-first-string "[" sep-string ", " sep-regexp ", +\\(and +\\)?" sep-regexp-submatch 0 post-last-string "]" ]]] [[[ The -string slots are used for writing; but what if you only have a regexp for the leading or trailing junk, but you want that restored exactly? You can set pre-first-string *after* the database file has been found. For instance, in db-before-read-hooks, use a function such as (defun btxdb:read-comments () (save-excursion (set-buffer db-buffer) (goto-char (point-min)) (if (search-forward "@" nil t) (sepinfo-set-pre-first-string (database-record-sepinfo database) (buffer-substring (point-min) (point)))))) or even put (sepinfo-set-post-last-string (database-record-sepinfo database) (save-excursion (set-buffer db-buffer) (goto-char (point-min)) (re-search-forward "\n\C-l\n") (buffer-substring (match-beginning 0) (point-max)))) as is in your auxiliary file. ]]] [[[ If all records have the same number of lines on disk, use the following function to return an appropriate sep-function. This is useful when, for instance, both the field separator and the record separator are the newline character. `make-n-line-sep-function' Return a sep-function useful when all records have exactly N lines on disk. ]]]  File: database.info, Node: Resolving ambiguities, Next: Problems with end-of-file newlines, Prev: How to specify regular file layouts, Up: Regular file layout Resolving ambiguities --------------------- Substitution and quoting are two mechanisms for dealing with the problem of distinguishing field and record separators from the contents of database records. For instance, if the newline character (actually, a string consisting of only the newline character) is used as a record separator, and records may contain multiline text fields (or other fields whose storage representation contains a newline), then how would EDB know, when reading the database back in, which newlines are record separators and which are part of fields? There are several ways to avoid this ambiguity. * Disallow the use in record fields of the character or string causing the ambiguity. For instance, in the example above, you might change the record field type of all of the string fields to one-line-string. * Change the separator(s) to strings that do not appear in the storage representation of any field. For instance, when reading a Unix password file, colons should not appear in the field text, so a colon can be used as the field separator, like so: (sepinfo-set-sep-string (database-field-sepinfo database) ":") Strings containing non-printing characters are another good bet, but this method relies on luck and the hope that the chosen separators will never appear in data. * Change the representation of the ambiguous string, when it appears in data; this guarantees that whenever the string does appear in a database file, it stands for a separator. This scheme is called substitution, because another string is substituted for the ambiguious one when it appears in data. This is similar to the previous workaround, which changed the separators rather than the data-bearing instances of the string. Ambiguities are still possible, if the substituted text happens to appear elsewhere in data. Specifying a substitution is described below. * When all of the previous methods are insufficient, the more powerful quoting mechanism can be used. It works similarly to the quoting mechanisms of programming languages that permit specification of strings which contain the character usually used to delimit string constants. It permits any strings to be used in separators and also to appear in data, but it slightly increases the size of the data and slows down reading a writing. Specifying that quoting be performed is described below. * The simplest solution is to use EDB's internal file layout (*note Internal file layout::.). Ambiguities can only occur when the field data and the separators are both text to be interpreted by EDB. EDB's internal file layout uses Emacs Lisp's mechanisms (a form of built-in quoting) to ensure that what is read in is identical to what was written out. The database designer need not worry further about the problem. [[[Describe substitution in detail.]]] Substitution is the replacement of potentially ambiguous strings by other ones. For instance, when writing tab-separated text, each occurrence of the newline character in a field could be replaced by control-k when the database is written. Then, when the file is read in, every newline can be safely assumed to be a record separator. The final step is converting the control-k characters back into newlines. This approach is taken by some marketed databases; for instance, I believe that FileMaker does just this. The problem with this approach is that if there were any control-k characters in the text, then when the database is read back in, they will be (incorrectly) converted to newlines. EDB warns when the database is being written out if this problem could occur; the user is given the option of choosing a different substitution or of aborting the database write operation. It is usually possible to find a substitution--a character or sequence of characters that doesn't appear in the data. [[[for instance, put eval: (database-set-substitutions database '(("\n" . "\C-k"))) in the "Local Variables" section of your format file. Or probably just put a short blurb here and put a longer one after the list.]]] [[[Describe quoting in detail.]]] Quoting is...  File: database.info, Node: Problems with end-of-file newlines, Prev: Resolving ambiguities, Up: Regular file layout Problems with end-of-file newlines ---------------------------------- Here is a subtle problem which can come up if you use `"\n\n"' as a record separator and exactly one newline appears at end of your database file. For convenience, EDB adds a record separator at the end of its working copy of the file, if there's not one already there. In this example, two newlines will be added, but then the file will still not end in a record separator, since after finding the first pair of newlines after the last record, EDB won't yet be at the end of the file because there will still be another character (namely, \n) there. The moral is that if there are any extra characters after the last record, even a newline, they should be specified. Either of the following forms will do the trick: (sepinfo-set-post-last-string (database-record-sepinfo database) "\n") (sepinfo-set-post-last-regexp (database-record-sepinfo database) "\n\\'") (sepinfo-set-post-last-regexp-submatch (database-record-sepinfo database) 0) The `\\'' is not strictly necessary in this example. [[[Maybe I should special-case this; i.e., replace end-of-file test with end-of-file-or-only-newline-remaining test? It comes up pretty frequently.]]] Here is an even more subtle problem: suppose that you want to get rid of every newline at the end of the database file, but you don't know how many there are. Using `"\n*\\'"' in place of `"\n\\'"' above will not work, because the post-last-record regexp is searched for backward from the end of the buffer, and (because of the way that `regexp-search-backward' is implemented) the backwards regexp match for `\n*' is always the empty string! The proper way to write this would be (sepinfo-set-post-last-regexp (database-record-sepinfo database) "[^\n]\\(\n*\\'\\)") (sepinfo-set-post-last-regexp-submatch (database-record-sepinfo database) 1)  File: database.info, Node: Tagged file layout, Next: Nonregular file layout, Prev: Regular file layout, Up: Database file layout Tagged file layout ================== Another popular file layout supported by EDB is that of field values preceded by the fieldname. For instance, a record might be represented in the file by Where:Here When: Now What: This! which indicates a record in which the `where', `when', and `here' fields have the specified values. Tagged files are a special case of files in nonregular layout; support for them is implemented through the mechanisms described in *Note Nonregular file layout::. To read a database file in tagged format, call the function `db-tagged-setup' in the database's format or auxiliary file. Its argument specifies the names of the fields and the tags that precede them in the database file. `db-tagged-setup' Ready the database to read files in tagged format. Creates database local variables and sets database slots. Argument TAGGED-FIELD-SPECS is a list of tagged-field specifications, one for each field in a database record. Each tagged-field specification is a three-element list of the field name (a symbol), the tag used to identify it in the file (a string), and a brief help string. Instead of a symbol, the tagged-field name may be a cons of the field name and its type. To indicate that a field is never found in the input file (typically because it is computed on the fly), use `nil' for its tag. This function should be called first in an auxiliary or format file, so that the defaults it chooses can be overridden. `database-set-fieldnames-to-list' should not be called if this function is. Calling this function sets the database's field names and installs appropriate functions for reading and writing the database. It also creates some database local variables (*note Local variables::.) which can be modified (by use of the `database-set-local' function) in order to customize the behavior of the parsing and output functions with respect to what characters can appear in a tag, what the separator between tag and value looks like, and how continuation lines are handled. By default, records are separated by blank lines, tags are separated from field values by `:', white space around the separator is not significant on input, the separator is followed by one tab on output, and continuation lines start with whitespace. `db-tagged-tag-chars' The characters that are allowed in field tags, in a form suitable for placing inside [] in a regular expression. `db-tagged-separator' The string that separates field names from values. Used only if db-tagged-separator-regexp or db-tagged-separator-output is `nil' (depending on whether the record is being read or written). `db-tagged-separator-regexp' A regexp for the separator between field names and values when parsing. `db-tagged-separator-output' The separator between field names and values on output. `db-tagged-continuation' The string that marks (the beginning of) a continuation line. Used only if db-tagged-continuation-regexp or db-tagged-continuation-output is `nil' (depending on whether the record is being read or written). `db-tagged-continuation-regexp' A regexp for a continuation line in a value when parsing. `db-tagged-continuation-output' The fixed string to use (before) continuing values on output. Other hooks permit arbitrary manipulations of records; for instance, if a database nearly conforms to the tagged file model, these can be used to customize the behavior of the existing tagged code. One way to do this is to have a function in `db-tagged-rrfr-hooks' remove the field from the file representation before the record is parsed, then have `db-tagged-wrfr-after-hooks' modify the automatically generated tagged file representation for that field. These functions can also be used for simpler tasks, of course. `db-tagged-rrfr-hooks' Hooks run on each database record before tagged parse. `db-tagged-wrfr-before-hooks' Hooks run before each tagged write of a database record. The record is bound to the dynamic variable record, and point is where the record will be inserted in the buffer. `db-tagged-wrfr-after-hooks' Hooks run after each tagged write of a database record. The record is bound to the dynamic variable record, and point is immediately after the file representation of the record.  File: database.info, Node: Nonregular file layout, Next: Reading from disk, Prev: Tagged file layout, Up: Database file layout Nonregular file layout ====================== Unlike many databases, EDB can work with data stored in any file layout whatever--so long as you specify how the information is to be extracted. If the file layout is too complicated to be described by regular expressions describing the record and field separators and their context (*note Regular file layout::.), then you may write Emacs Lisp code which extracts the information from the database file. The great advantage of this mechanism is that it permits you to maintain your current files, in exactly their current file layouts, and to keep the same tools and habits you've accumulated with respect to them, but also to manipulate them in a structured way with EDB when necessary. For instance, you might wish to maintain the database file in a file format easy for people to read all the time, rather than having to create a report for that purpose. Three pieces of information must be provided: how to find the extent of a file record, how to read a file record, and how to write a file record. The third may be omitted if the database is only being read in the custom file layout (and will be saved in some more tractable file format). If the second is provided (that is, the `read-record-from-region' database slot is set), then the file will be assumed to be in a nonregular file layout and the value of that slot will be used to read the database, no matter what other information is provided. Information about how to separate one record from another within the file is found in the `record-sepinfo' slot of the database, as usual. In many cases, even if the file layout of the data is nonregular, it will be easiest to describe the record separator with a string or a regexp. For more details, see *Note Regular file layout::. You may also set the sepinfo's `sep-function' slot to a function. The function should take one argument, the end of the previous record (`nil' the first time it's called), and return a pair whose car is the end of the current record and whose cdr is the start of the next record (`nil' if there is no next record in the file). The `read-record-from-region' slot of the database contains a function of no arguments which, when called with the current buffer narrowed to a single file record (that is, narrowed to the representation of a single database record), returns a record in the database's internal file layout. The variable `database' is dynamically bound to the current database, and so the right way to create the record to be returned is via `(make-record database)'. Its fields can then be set with `record-set-field'. The `write-region-from-record' slot of the database optionally contains a function which takes a database record as its argument and inserts the file representation of that record in the current buffer; the variable `database' is dynamically bound to the current database. If this slot is not specified (and slot `internal-file-layout-p' is `nil'), then the fieldsep and recordsep information, if present, will be used to write the record (*note Regular file layout::.). This permits the use of a simple, regular output file layout with a more flexible input file layout. Tagged format is a special case of nonregular file layout for which EDB provides support; see the implementation of support for tagged database files in `db-tagged.el' and *Note Tagged file layout::. Another example is given below. * Menu: * Nonregular database example:: Example of database in nonregular file layout  File: database.info, Node: Nonregular database example, Up: Nonregular file layout Example of database in nonregular file layout --------------------------------------------- Here is a simple example of a database in a nonregular file layout; this does not mean that the file representation of each record is vastly different from the others (it may be, but is not in this instance), but that there is no regular rule for extracting field values from the record. Suppose we had a database of the following form: Place: Dentist's Office Time: Never! Purpose: Root canal Place: Home Time: Midnight Purpose: Sleep Place: Other places Time: Other times Purpose: Other things Then in order to read and write this database, we place the following code in the auxiliary file (*note Reading from disk::.): (sepinfo-set-sep-string (database-record-sepinfo database) "\n\n") (database-set-read-record-from-region database 'arb-demo-rrfr) (database-set-write-region-from-record database 'arb-demo-wrfr) (defun arb-demo-rrfr () (goto-char (point-min)) (if (re-search-forward "Place:[ \t]*\\(.*\\)\nTime:[ \t]*\\(.*\\)\nPurpose:[ \t]*\\(.*\\)") (let ((result-record (make-record database))) (record-set-field result-record 'place (match-string 1) database) (record-set-field result-record 'time (match-string 2) database) (record-set-field result-record 'purpose (match-string 3) database) result-record) (error "This didn't look right to me."))) (defun arb-demo-wrfr (record) (insert "Place: " (record-field record 'place database) "\nTime: " (record-field record 'time database) "\nPurpose: " (record-field record 'purpose database))) The auxiliary file would also specify the database's fieldnames: (database-set-fieldnames-to-list database '(place time purpose)) as well as possibly other information such as the summary format or the name of the default format file. See the example database auxiliary file `arb-demo.dba' for a concrete example of this. All this Emacs Lisp code may be placed in "Local Variables" section of the format file instead of in the auxiliary file, if desired. For more information about the "Local Variables" section of a file, *Note Variables: (emacs)File Variables. This particular example is simple enough that a special function for reading isn't strictly necessary. Reading can be done under the control of regular expressions; for instance, each field separator would be `"\n[^:]*:[ \t]*"'. See the example database auxiliary file `arb-demo-regexp.dba' for a concrete example of this. Or, you could just use EDB's support for database files in tagged file layout, which is exactly what this is; see *Note Tagged file layout::.