Introduction
************

The dtype++ library provides a simple and space-efficient interface between C++ and Scheme programs and data structures.  It provides a way to create, manipulate, and evaluate Scheme-like data structures from C++.  It is reasonably space-efficient, and time-efficient where that is possible without compormising the cleanliness of its interface.

There are several layers of function provided by the dtype++ library.  At the lowest level, dtype++ provides a coherent way to manipulate both fundamental and user-defined types into pairs, lists, vectors, sets, and associative arrays.  The library also provides a run-time type system and a built-in garbage collector with both reference-counting and stop-and-copy semantics. (This doesn't actually work yet.)  Finally, dtype++ provides a simple and embeddable Scheme interpreter, with provisions for network-transparent RPC servers.

The dtype++ library is not, however, designed, to be a production-quality Scheme programming environment.  In particular, it does not provide a byte-compiler or microcode-based interpreter.  The dtype++ library uses the C++ stack for function calls, and many of the more advanced Scheme features are provided in and often inefficient ways.  It is assumed that the Scheme interpreter will be used primarily for higher-level scripting functions. Since the interface between Scheme and C++ is so clean, I expect that most low-level functions will be implemented directly in C++ for efficiency.

Overview
********

A dtype is a single data object which can hold different types of data. Some dtypes are simple values, such as integers and floating point values.  Other dtypes store more complicated values, such as strings, lists, vectors, and packets of raw data bytes.

Even though a dtype might be one of several types, a dtype's type is permanently fixed once it is created.  A dtype can only contain values of the type for which it was created, and it may contain only one value at a time.

Once data is in the form of any of these dtypes, any of the dtype routines may be applied to that data.  For example, there are routines for writing dtypes to disk, or transmitting dtypes across the network.

In addition to the routines which work for all dtypes, there are routines that are specific to each kind of dtype.  For example, there is a large library of routines for manipulating the dt_vector dtype, which includes operations for inserting and deleting elements, and for concatenating multiple dt_vectors.

dtype types
===========

There are eight different types of dtypes.  They are:

dt_null
     A dtype with no value.

dt_int
     A signed integer of fixed but unspecified length.

dt_real
     A floating-point number of unspecified range and precision.

dt_packet
     A buffer of raw bytes and a length field.  Also used for storing data that does not readily fit any of the other dtype categories, such as digitized images or sound data.

dt_string
     A variable length, zero-terminated ASCII string.

dt_symbol
     A variable length, zero-terminated ASCII string.

dt_vector
     A variable length, ordered list of dtypes.

dtype ASCII representations
===========================

There many functions for manipulating dtypes with input and output. Every dtype has a human-readable ASCII representation, and dtypes may be written to a file, written to a string, or printed to a stream in this ASCII form.  There is also a parser for dtypes, which allows the ASCII representation to be translated back into a dtype.  This makes it easy for a program to store its dtypes in a human-readable form that can be edited by a standard text editor.  This also makes it easy users to enter dtype data into a running program.  As a side note, files which contain ASCII representations of dtypes should have the extension .dtype.

The following is an example of the ASCII representation of a dt_vector dtype:

     #(3 "a string!" #(-87 3))

This is a single dtype, but since it is a dt_vector it holds several dtypes within it.  In this case, the dt_vector holds 4 elements of various kinds --- integers, and strings.  One of the elements is itself a dt_vector which contains 2 integer elements.  This is the form in which a dtype might be printed out, and this ASCII string could also be turned back into a dtype by the parser.

dtype Binary representations
============================

In addition to input and output through ASCII representations, dtypes can also use a binary representation which is usually more compact and faster to read in and write out than the ASCII representation.  When communicating dtypes across the network, this binary form is used almost exclusively.  The drawback to the binary representation is that files written in this form are not human-readable.  Files which contain binary representations of dtypes should have the extension .dtbin.

dtype Environments
==================

dtypes are a convenient way of expressing a list of keyword/value pairs , such as the following:

     (
      ("name" "john")
      ("address" (38 "Hopeless Ave."))
      ("idnumber" 367552)
      ("interests" ("hockey" "music" "farming"))
     )

This single dtype describes a person's profile as a list of keyword/value pairs.  Such a list is called a dtype environment. The dtype library contains routines for manipulating dtype environments, such as routines which find the value associated with a certain keyword, or routines which add or remove keyword/value pairs from the environment.

The keyword portion of each keyword/value pair is usually a string, but is permitted to be any kind of dtype.

dtype Evaluator
===============

In addition to storing data, dtypes can also be used to store programming code.  This is especially useful because dtypes are easy to transfer across the network between different processes, which means that processes can dynamically load programming code or send programs to other processes.

In order for programs to be represented as dtypes, the programs must be written in a special programming language called DEVAL, for dtype EVALuator.  This language has a very simple syntax which is designed especially for dtype structures, a syntax which is based on LISP.  The following is an example of a DEVAL statement:

     ("+" 3 4 ("*" 2 4))

This DEVAL statement is a normal dtype, but it is in the form of the DEVAL language.

The dtype library supports the DEVAL language by providing a dtype Evaluator, which is an interpreter for DEVAL statements.  For example, feeding the above DEVAL statement into the dtype Evaluator will produce a result of 15.

The dtype Evaluator is designed to be as small and simple as possible, which means that the DEVAL language is also extremely "bare-bones". The library of DEVAL operations is very basic, consisting mostly of arithmetic operators, looping statements, and some provision for variables.  The real power of DEVAL is its expandability.  New application-specific operations can be written and added to the DEVAL language, which means that an application can custom fit the DEVAL language to its own needs.

dtype Servers
=============

dtypes are an especially nice way for network clients and servers to communicate.  Using flexible dtype structures makes it easy to design and implement a wide variety of high-level communication protocols.  It also allows the protocol to be expanded while still retaining backward compatibility with older protocols.

The dtype library contains some support for writing dtype servers and clients.  More importantly, there is an example model of a dtype server/client pair, which can be taken and modified for each particular application.

Creating and using dtypes
*************************

The dtype library is written in C++, which means that any programs using dtypes must also be compiled with a C++ compiler.  A program which uses dtypes must also include the following statement:

     #include <dtype++/dtype++.hh>

When linking a program using dtypes, the following libraries must be linked in:

     -ldtype++ -ldsys++ -lm

There are three major aspects to using dtypes: creating dtypes, manipulating dtypes, and destroying dtypes.

Creating and Destroying dtypes
==============================

dtypes, like all C++ objects, are created using constructors.  A constructor will create a dtype object, set the type of the dtype object, and initialize the object with a value.  Each of the eight dtype types has its own constructor.  For example:

     dt_int dt(6);

is the code used to declare a dtype Integer object called DT, and initialize its value to 6.  Some dtypes, such as dt_null and dt_bang, do not need to be initialized with a value.  The constructors for these dtypes would look like the following:

     dt_bang dtb;

Note that if the constructor takes no values, then there should be NO parentheses after the dtb.

Using dtypes
============

Using a created dtype is a matter of invoking the various methods and operators that are provided for that dtype.  Each type of dtype has its own set of methods and operators that are specific to the dtype type.  In addition, there are certain methods that are common to all dtype types.

How to invoke dtype methods
---------------------------

Invoking a dtype's method is as simple as following the dtype's pointer with an arrow (->) and the method's name.  For example, if the dtype DTL is a dt_vector *, then the following expression:

     dtl->length()

will return the number of elements in that list.  NOTE: it is a common mistake to forget the parentheses.  Be careful.

Some methods will take arguments.  For example, if DTI is a dt_int *, then the following statement:

     dti->set(7);

will set the value of dti to 7.

If the object is not a dtype pointer but is an actual dtype object or a reference to a dtype object, then the methods can be invoked by replacing the -> operator with the . operator.  However, just about all dtype routines return pointers to dtypes rather than dtype references.

How to invoke dtype operators
-----------------------------

Certain dtype functions can be accessed using operators.  NOTE: operators will only work on dtypes or dtype references, NOT dtype pointers.  If you have a dtype pointer, you can convert it to a dtype by dereferencing it with the * operator.  For example, if DT is a dtype *, then (*dt) is a dtype, suitable for use with operators.

Some operators are redefined as a shorthand for specific dtype methods. For example, the [] operator is a shorthand for the elem operator.  If DTL is a dt_vector, then the expression:

     (*dtl)[5]

is equivalent to:

     dtl->elem(5)

Both will return the 5th element of the list.

Certain operators are also redefined to work for all dtypes, such as the relational/equality operators (==, !=, <, >, etc.).

dtype class hierarchy
*********************

dtype is the virtual base class for all dtypes.  dtype has eight subclasses, representing the eight different types of dtypes. These subclasses are:

dt_null
     The NULL dtype class

dt_bang
     The bang dtype class

dt_int
     The integer dtype class

dt_real
     The floating-point dtype class

dt_string
     The string dtype class

dt_vector
     The list dtype class

dt_packet
     The data packet dtype class

dt_ref
     The list reference dtype class

All functions which are available for any of the dtype subclasses are also defined virtually for the dtype base class.  Which means that the methods for a subclass can be invoked on an object which is cast as the base dtype class.  For example, consider the following code:

     dt_vector *dtl;
     dtype *dt;
     int len;

     ...
     dt = dtl[3];
     len = dt->length();

The first statement assigns DT to a pointer to the 3rd element of DTL.  The second statement assumes that that element is itself a list and extracts the length of that list.  Note that length() is an operation reserved for a dt_vector, not for any dtype, but even though DT is a pointer to a dtype and not a dt_vector, the system will figure out that DT is actually pointing to a dt_vector and correctly invoke length() on that list.  If DT were pointing to some other kind of dtype, then a runtime error would occur.

Common dtype routines
*********************

The dtype library contains routines that can be used on dtypes of any type.  They fall into the following categories:

Determining the type of a dtype
===============================

The following methods deal with determining the type of a dtype.

dt.type()
     Returns an integer representing what kind of dtype dt is.  The returned value will be one of the following: DT_NULL, DT_INT, DT_REAL, DT_STRING, DT_BANG, DT_PACKET, DT_VECTOR, DT_REF.  If 0 is returned, then dt is not a valid dtype.

dt.typestr()
     Returns a pointer to a string which describes what kind of dtype dt is. The returned value will be one of the following: "null", "int", "real", "string", "bang", "packet", "list", "ref".

ASCII printing and parsing of dtypes
====================================

The following methods and operators deal with the printing of dtypes to an ASCII representation.

print(ostream &pos)
     Prints an ASCII representation of the dtype to stream pos.  The representation is always followed by a space, but not necessarily a carriage return.

ostream &pos << dt
     Also prints an ASCII representation of the dtype to stream pos, and returns pos.  This allows dtypes to be printed out just like any other object.  For example:

          cout << "The dtype is: " << dt << endl;

     will print out a dtype to stdout.  NOTE:  dt may be a dtype, or a pointer to a dtype.

There are a few routines for parsing a dtype from an ASCII representation of a dtype.  These routines return new dtypes that the programmer is responsible for deleting.

dtype::parse(istream &pis, [int *ateof])
     Parses one dtype from the stream pis.  The dtype is created and a pointer to it is returned.  For example, the following code parses dtypes from stdin:

          dtype *dt;

          while (1) {
          	cout << "Enter a dtype" << endl;
          	dt = dtype::parse(cin);
          	cout << "You entered " << dt << endl;
          	delete dt;
          }

     Ateof is an optional argument which can be used to detect the end of file.  If supplied, it must point to an integer.  If dtype::parse attempts to read past the end of the file, then it will return NULL, and ateof will be filled with the value 1, otherwise ateof is filled with the value 0.  If an improperly formatted dtype was entered, then dtype::parse will return NULL, but ateof will be filled with 0.

dtype::parse(char *pstr, [int *ateof])
     Parses one dtype from a string pstr.  The dtype is created and a pointer to it is returned.  Ateof behaves the same as above.

dtype::parse(int pf, [int *ateof])
     Parses one dtype from a file descriptor pf.  The dtype is created and a pointer to it is returned.  Ateof behaves the same as above.

dtype::parse(FILE *pf, [int *ateof])
     Parses one dtype from a FILE pf.  The dtype is created and a pointer to it is returned.  Ateof behaves the same as above.

Reading and writing binary dtype representation
===============================================

There are also several routines for writing out the binary representation of a dtype (assume DT is a dtype *).

dt->write(ostream &pos)
     Writes the binary representation of dt to the stream pos.  Returns 0 if successful, -1 if an error occurred.

dt->write(int pf)
     Writes the binary representation of dt to the file whose file descriptor is pf.  Returns 0 if successful, -1 if an error occurred.

dt->write(FILE *pf)
     Writes the binary representation of dt to the FILE whose FILE pointer is pf.  Returns 0 if successful, -1 if an error occurred.

dt->write(void *pbuf)
     Writes the binary representation of dt to an area in memory pointed to by pbuf.  Returns 0 if successful, -1 if an error occurred.

dt->write(dsys_bufclient *pdsbc)
     Writes the binary representation of dt to the network through a dsys_bufclient pdsbc.  If the dsys_bufclient is attached to a server, then the dtype will emerge intact at the server on the other end of the network.

Each write routine also has a corresponding read routine.  These read routines create new dtypes from scratch, so they are not directly associated with any class.  The programmer is responsible for deleting the dtypes that are created by these routines.  These routines are:

dtype::read(istream &pis)
     Reads the binary representation of a dtype from the stream pis.  The dtype read in is created and a pointer to it is returned.

dtype::read(int pf)
     Reads the binary representation of a dtype from the file with file descriptor pf.  The dtype read in is created and a pointer to it is returned.

dtype::read(FILE *pf)
     Reads the binary representation of a dtype from the FILE with FILE pointer pf.  The dtype read in is created and a pointer to it is returned.

dtype::read(void *pbuf)
     Reads the binary representation of a dtype from a memory buffer pointed to by pbuf.  The dtype read in is created and a pointer to it is returned.

dtype::read(dsys_bufclient *pdsbc)
     Reads the binary representation of a dtype from the network through a dsys_bufclient pdsbc.  The dtype must have been sent through a corresponding write to a dsys_bufclient on the other side of the network. The dtype read in is created and a pointer to it is returned.

dtype comparisons
=================

dtypes may be compared against each other to determine if one dtype is less than, equal to, or greater than another dtype.

dtype comparison rules
----------------------

If two different kinds of dtypes are compared against each other, then their ordering is determined only by their types.  The order of types is as follows:

     dt_null
     dt_int
     dt_real
     dt_string
     dt_bang
     dt_packet
     dt_vector
     dt_ref

Thus any dt_vector will be considered greater than any dt_string.  No attempts are made to coerce one type into another, which means that a dt_real will always be greater than a dt_int, no matter what their values are.

If two dtypes of the same type are compared against each other, then the result will depend on what that type is:

Null
     Two Null dtypes are always equal.

Bang
     Two Bang dtypes are always equal.

Int
     The values of the two dt_ints are compared.

Real
     The values of the two dt_reals are compared.

String
     The values of the two dt_strings are lexically compared.

List
     The elements of the two dt_vectors are compared one at a time in order, until one pair of elements is found to differ.  If one of the dt_vectors runs out of elements, then that is considered to be the smaller dt_vector.

Packet
     The buffer sizes are compared first, with the smaller buffer size resulting in the smaller dt_packet.  Then the data in the two dt_packets are compared, one byte at a time.  If any bytes differ, then the smaller byte represents the smaller dt_packet.  Finally, the "fields" dtypes are compared.

Ref
     The number of indices are compared first.  Then the indices are compared in order one at a time.  Tail indices are considered greater than normal indices.

dtype comparison methods
------------------------

There is one general comparison method for dtypes (assume DT1 and DT2 are both dtype *).

dt1->compare(dt2)
     Returns a negative number if dt1 is less than dt2, 0 if dt1 is equal to dt2, and a positive number if dt1 is greater than dt2.

There are several shorthand operators for doing specific dtype comparisons.  Note that these operators expect dtypes or dtype references, not pointers to dtypes.

(*dt1) < (*dt2)
     Returns 1 if dt1 is less than dt2, 0 otherwise

(*dt1) <= (*dt2)
     Returns 1 if dt1 is less than or equal to dt2, 0 otherwise

(*dt1) > (*dt2)
     Returns 1 if dt1 is greater than dt2, 0 otherwise

(*dt1) >= (*dt2)
     Returns 1 if dt1 is greater than or equal to dt2, 0 otherwise

(*dt1) == (*dt2)
     Returns 1 if dt1 is equal to dt2, 0 otherwise

(*dt1) != (*dt2)
     Returns 1 if dt1 is not equal to dt2, 0 otherwise

dtype copying
=============

There are several routines available for copying one dtype to another (assume DT and DT2 are both type dtype *).

dt->copy()
     Creates and returns a reference to a new dtype which is a "shallow copy" of dt.  This means that for compound types (strings, lists, packets, and references), dt and its copy will both point to the same data.  Thus, manipulations done on the data in dt will be reflected in its copy, and if dt is destroyed, then the data in dt's copy will be invalid.  Be careful not to get burned by this.

dt->copyR()
     Creates and returns a reference to a new dtype which is a full copy of dt.  All the data in dt is recursively duplicated and assigned to the copy.  The data in the copy is thus completely separated from the data in dt, and there will be no problems with the two dtypes pointing to the same data.  The disadvantage to this routine over the regular copy is that it takes longer and results in doubling the amount of memory used by the data in dt.

dtype constructors, operators, and methods
******************************************

The following outlines the constructors and methods that are specific to each of the eight dtypes.

NULL dtype
==========

The dt_null type represents a dtype with no value.  It has a single constructor:

dt_null dt
     Declares a dt_null dtype.

Signal dtype
============

The dt_bang type represents a dtype with no value.  It is usually used to represent a signal with no value when passed between processes. It has a single constructor:

dt_bang dt
     Declares a dt_bang dtype.

Integer
=======

The dt_int type represents a 32-bit signed integer value.  It has a single constructor:

dt_int dt(long pval)
     Declares a dt_int with the value pval.

The following functions and operators are provided (assume DT is a dt_int *).

dt->ival()
     Returns the integer value in dt.

dt->set(long pval)
     Sets the integer value in dt to pval and returns a pointer to dt.

Floating point
==============

The dt_real type represents a 4 byte floating point value.  It has a single constructor:

dt_real dt(double pval)
     Declares a dt_real with the value pval.

The following functions and operators are provided (assume DT is a dt_real *).

dt->dval()
     Returns the floating point value in dt as a double.

dt->fval()
     Returns the floating point value in dt as a float.

dt->set(double pval)
     Sets the floating point value in dt to pval and returns a pointer to dt.

Strings
=======

A dt_string type represents a zero-terminated ASCII string.  The memory for the string is allocated and maintained by the dt_string.  When the dt_string is destroyed, the memory allocated for that string is also destroyed.

dt_string has the following constructors:

dt_string dt(char *pval)
     Declares a dt_string with the string value taken from pval.  The dt_string actually allocates its own string space and copies the data from pval into that space, so that the dt_string does not depend on the data in pval once the dt_string is created.

dt_string dt(int size)
     Declares a dt_string which has enough space to hold a string of (size - 1) characters (the 0 terminator takes one byte).  The string data is completely uninitialized --- it may be filled in later.

dt_string dt
     Declares a dt_string which is completely uninitialized, without even a buffer allocated to hold a string.  The "set" method must be used to initialize a dt_string constructed this way.

The following functions and operators are provided (assume DT is a dt_string *).

dt->cval()
     Returns a char * pointer to the string value contained in dt.

dt->strlen()
     Returns the number of characters in dt's string.  Technically, this returns the size of dt's buffer minus 1 (for the zero-terminator).

dt->set(char *pval)
     Sets the string value of dt to pval, by allocating a char buffer big enough to hold the string in pval, and then copying the string from pval to that buffer.  If there already was a string in dt, then that original string is lost and will not be destroyed when dt is destroyed.  This routine should be used with caution, usually only in combination with the completely uninitialized version of the constructor.  Returns a reference to dt.

Variable-length lists
=====================

A dt_vector represents a variable-length ordered collection of dtypes, which are called the dt_vector's elements.  In general, a dtype should not be manipulated as a member of more than one list, unless very special care is taken.  When a dt_vector is destroyed, all of its elements are also destroyed.

dt_vector has the following constructors:

dt_vector dt
     Declares an empty dt_vector

dt_vector dt(dtype *dt1, dtype *dt2, ... , NULL)
     Declares a dt_vector which contains the specified dtypes in it.  NOTE: DO NOT LEAVE OUT THE "NULL" AT THE END OF THE LIST.

     As a quick example, look at the following code:

          dt_vector *dt = new dt_vector(new dt_int(3),
          			new dt_string("hello"),
          			new dt_real(4.5),
          			NULL);

     The result of this is that dt points to (3 "hello" 4.5).

The following functions and operators are provided (assume that DT and DTL are both dt_vector *, and that DTA and DTAT are dtype *.

dt->length()
     Returns the number of elements in the list.

dt->elem(pelem)
     Returns a pointer to the pelem'th element of the list.  The first element of the list is numbered 0.  The list can also be referenced from its end, by using negative numbers.  For example, element -1 is the last element of dt, -2 is the second to last, etc.

     This routine works fastest if you access the element which is the same or one greater than the element you last accessed.

dt->tailfrom(pelem)
     Returns a new dt_vector which contains all the elements from dt starting from element number pelem.  Note that dt and this new list will share the same elements, which could lead to confusion.  Be careful.

dt->append(dta)
     Appends a single dtype dta to the end of dt, and returns dt.  Note that "appending" is different from "concatenating".  For example:

          (3 4 5) append "hello"  ->  (3 4 5 "hello")
          (3 4 5) append (6 7 8)  ->  (3 4 5 (6 7 8))  NOT  (3 4 5 6 7 8)

dt->prepend(dta)
     Inserts a single dtype dta to the beginning of dt, and returns dt.  This also follows the rule of "appending", not "concatenating".

dt->concat(dtl)
     Concatenates a single dt_vector dtl to the end of dt, and returns dt.  dtl must be a dt_vector.  Note that "concatenating" is different from "appending".  For example:

          (3 4 5) concat (6 7 8)  ->  (3 4 5 6 7 8)  NOT  (3 4 5 (6 7 8))

dt->remove(dta)
     Removes a single dtype from the list.  dta must be an element in dt.  Also note that the removed element is not automatically destroyed.  If you want to automatically destroy the removed element, then specify DT_FREE as the second argument.

     For example, to remove and destroy the second element of the list:

          dt->remove(dt->elem(2), DT_FREE);

dt->insert(dta, dtbefore)
     Inserts a single dtype dta into dt, before element dtbefore, and returns dt.  dtbefore must be an element in the list.  For example, to insert an "inserted" before the second element of the list:

              before: (1 2 3 4)
          dt->insert(new dt_string("inserted"), dt->elem(2));
              after: (1 2 "inserted" 3 4)

dt->replace(dtnew, dta)
     Replaces the value dta in dt_vector dt with dtnew.  The old value dta is not automatically destroyed.  If you want to automatically destroy the replaced element, specify DT_FREE as the third argument.  For example, to replace and destroy the second element of a dt_vector:

            before: (1 2 3 4)
          dt->replace(new dt_string("replaced"), dt->elem(2), DT_FREE);
            after: (1 2 "replaced" 4)

dt->split(dta)
     "Splits" the dt_vector into two separate lists at element dta.  dta must be an element in dt.  The first of the two lists starts at the beginning of the original list and goes to the element before dta.  The second of the two lists starts at dta and goes to the end of the original list.  dt is modified to hold the first list.  The second list is returned as the result of the expression.  For example:

            before: dt = (1 2 3 4 5)
          dt_vector *dt2 = dt->split(dt->elem(2));
            after: dt = (1 2)  dt2 = (3 4 5)

dt->detach()
     "Detaches" a dt_vector from its elements, without destroying the elements. This is usually used when two dt_vectors are concatenated together.  The first dt_vector is modified to hold the entire list, but the second dt_vector still points to part of that list.  If you try to free the memory used by the second dt_vector, as you should, those elements will disappear and the first dt_vector will be corrupted.  The solution is to "detach()" the second dt_vector, thus detaching it from its elements, then delete it.

            before: dt1 = (1 2 3)  dt2 = (4 5 6)
          dt1->concat(dt2);
            now: dt1 = (1 2 3 4 5 6), but dt2 still = (4 5 6).  If you free
            dt2 now, dt1 = (1 2 3 GaRbAgE GaRbAgE...)
          dt2->detach();
            now you can safely delete dt2.
          delete dt2.

In addition to the above methods, the following operators provide a shorthand for some of these methods.  Remember that these operators will only work on dt_vectors or dtype references, not pointers.

(*dt)[pelem]
     Equivalent to dt->elem(pelem).

(*dt) += dta
     Equivalent to dt->append(dta).

(*dt) -= dta
     Equivalent to dt->remove(dta).

Data packet
===========

A dt_packet is a dtype which is used to hold a buffer of "raw" bytes.  "Raw" means that the bytes are not interpreted in any way by the dtype library.  The dt_packet is merely a way to hold that data and pass it around.

A dt_packet consists of a pointer to a buffer, and a fields dtype.  This dtype usually contains a dtype environment (see the section on dtype environments) which holds information about the data. For example, a dt_packet might hold a digitized image, and would use the fields dtype to store the image's height, width, color lookup table, etc.

When a dt_packet is destroyed, its associated data buffer and the fields dtype are also destroyed.

dt_packet has the following constructors:

dt_packet dt
     Declares an empty dt_packet with no buffer or fields dtype.  Should rarely be used.

dt_packet dt(long pbufsize);
     Declares a dt_packet with a buffer of pbufsize bytes.  The buffer is automatically allocated, but is not initialized to any particular value. The fields dtype is initialized to be an empty list (which is an empty dtype environment).

dt_packet dt(const char *pfname);
     Declares a dt_packet which is initialized to hold the data from the file specified by pfname.  The buffer is allocated to be exactly the size of the file, and the data from the file is read into the buffer.  The fields dtype is initialized to be an empty list (which is an empty dtype environment).

The following functions and operators are provided (assume DT is a dt_packet *)

dt->bufsize()
     Returns the size of the buffer, in bytes.

dt->fields()
     Returns a pointer to the fields dtype in the dt_packet.

dt->vval()
     Returns a void * pointer to the actual data buffer in the dt_packet.

dt->set(void *pval, long pbufsize)
     Sets the buffer to be pval, and the buffer size to be pbufsize.  This should be carefully, if at all, because if a buffer was already allocated, it will be lost (but its memory will NOT be freed).  Usually, this will only be used in conjunction with the empty dt_packet constructor or after a dt_packet has been detached.  Returns dt.

dt->detach()
     Disconnects a dt_packet from its data.  This allows a program to "pull the data" out of a dt_packet, so that if the dt_packet is freed, the data is not freed with it.  Returns dt.

dt->set_fields(dtype *pfields)
     Sets the fields dtype to be pfields.  The same warnings apply here as they do for set().  It is usually preferable to avoid using this function by allowing the constructor to create the empty fields dtype and have the programmer modify, rather than replace, that dtype.

List reference
==============

A dt_ref is best described as a general "pointer" into dtype lists.  For example, a dt_ref can hold a value which represents "the third element of the second element of the list".  When that dt_ref is applied to a dt_vector, the result will be the third element of the second element of that list.

A dt_ref is organized as an array of indices.  For the example above, the dt_ref would be an array of two indices, with the values 2 and 3.  When the dt_ref is applied to a dt_vector, the indices are applied in order.  In this example, the 2 index would be applied first, thus getting the second element of the list, then the 3 index would be applied, which would get the third element of that list.  Thus the end result would be the third element of the second element of the list.

A dt_ref has a special ASCII representation, which is simply the list of indices separated by periods.  A dt_ref representation always begins with an @ sign.  So, the representation of the above example would be:

     @2.3

A dt_ref is also allowed to be completely empty, represented by only an @ sign.  Applying an empty dt_ref to a dtype returns that dtype.

A dt_ref can also be used to extract a sublist from a list, rather than a single element.  For example, an index can mean "the rest of the list starting at the third element" rather than "the third element of the list".  These kinds of indices are called "tail indices".  Their ASCII representation is the same as normal indices, but they are preceded by a comma rather than a period.  For example:

     @2,3

means "the rest of the list starting at the third element of the second element of the list."  If applied to the following list:

     (0 1 (10 11 12 13 14 15) 3 4)

the result will be:

     (13 14 15)

As another example:

     @,2

means "the rest of the list starting at the second element".  When applied to:

     (0 1 2 3 4 5 6)

the result is:

     (2 3 4 5 6)

When a dt_ref is applied, its indices are applied in order.  If at any point, the dtype that the index is being applied to is not a list, then that dtype is immediately returned.  For example:

     @2,3 applied to (0 1 2 3 4) returns 2, because 2 is not a list.

Applying a dt_ref to a dtype does not modify that dtype in any way.

dt_ref has the following constructors:

dt_ref dt
     Declares an empty dt_ref.

dt_ref dt(const char *pstr)
     Declares a dt_ref whose value is taken from the ASCII value in pstr.  For example:

          dt_ref *dt = new dt_ref("@2.3");

     creates a new dt_ref which means "the third element of the second element of the list".

The following functions and operators are provided (assume declaration of dt_ref dt = "@2,3").

dt->index(int pi)
     Returns the value of index number pi.  For example, dt->index(0) == 2, and dt->index(1) == 3.

dt->istail(int pi)
     Returns whether or not index number pi is a tail index.  For example, dt->istail(0) == 0, dt->istail(1) == 1.

dt->add_index(int pi)
     Adds a new index to the end of dt, and returns dt.  For example, dt->add_index(4) returns the dt_ref "@2,3.4".

dt->add_tailindex(int pi)
     Adds a new tail index to the end of dt, and returns dt.  For example, dt->add_tailindex(4) returns the dt_ref "@2,3,4".

dt->apply(dtype *dta)
     Applies the dt_ref to the dtype dta and returns the result.

ASCII dtype representations
***************************

This section details the ASCII representations of each of the dtypes. These are the representations that are used the print out a dtype, and these are the representations that are recognized by the dtype parser.

dt_null ASCII representation
============================

The ASCII representation of a dt_null is the string NULL.

dt_bang ASCII representation
============================

The ASCII representation of a dt_bang is the string bang.

dt_int ASCII representation
===========================

A dt_int is represented in ASCII the same way that C prints out and parses integers.

dt_real ASCII representation
============================

A dt_real is represented in ASCII the same way that C prints out and parses floating point values.

dt_string ASCII representation
==============================

The ASCII representation of a dt_string is the value of the string, surrounded by quotation marks.

C-style escaping (\n, for example) will be parsed correctly into a dt_string.  Currently, the only escape characters allowed are \" (quote), \\ (backslash), \n (newline), \t (tab), \b (backspace).  These escape commands are translated by dtype::parse into the correct characters before being placed into the string, so if the user entered a \b in the middle of a string, that would be translated into the value 8 before being put into the string.  Quotation marks and backslashes MUST be entered as \" and \\ to the parser (the parser will translate them simply into " and \).

To enter a string into the parser, surround the string with quotation marks and make sure that you escape any quotation marks and backslashes. One useful fact that you can take advantage of is that any string of alphanumeric characters is automatically treated as a dt_string even without the surrounding quotation marks, ONLY IF IT DOESN'T PARSE INTO SOME OTHER dtype FIRST.  So, hello2u will be recognized as a dt_string, 25 will be recognized as a dt_int, but "25" will be recognized as a dt_string.  If you want to be sure, surround with quotation marks.

There is an alternate method of entering a string as straight hexadecimal bytes.  To do this, enter the hexadecimal values, two characters each not separated by spaces, and surrounded by < and >.  This method will not automatically place a terminating zero at the end --- if you want one, enter it yourself.  For example, the following:

     <48656C6C6F00>

would parse into the string "Hello".

dt_vector ASCII representation
==============================

A dt_vector is represented as the ASCII representations of the elements, surrounded by parentheses.  For example,

     (1 2 3 )

represents a list of three dt_int elements.

dt_packet ASCII representation
==============================

The ASCII representation of a dt_packet consists of three parts: the size of the dt_packet, the fields dtype in the dt_packet, and the value of the data in the packet.  The format is as follows:

     [<size> <fields> <value>]

The <size> is an integer, the <fields> is printed as a normal dtype, and the <value> is printed out in the hexadecimal string form (see the ASCII representation of dt_strings).  The following is an example:

     [10 (("width" 5 ) ("height" 2 ) ) <00010203040506070809>]

A dt_packet can be entered to the parser in this format.  The value portion can also be entered in the normal "string" format (which will add a terminating 0 to the end).  All the values do not have to be specified, and in fact the value field doesn't have to be specified at all.  Any values which are not specified are left undefined.

There is an alternate method for entering a dt_packet, which allows the dt_packet to be entered from a file.  The format for this is:

     ["<filename>" <fields>]

When the parser encounters this, it will create a dt_packet the size of the specified filename, then read the file as the value of the dt_packet.

dt_ref ASCII representation
===========================

The ASCII format of a dt_ref is as follows:

     @<index><index><index>...

where each <index> is a number preceded by either a period or a comma.  An index preceded by a period represents a normal index, an index preceded by a comma represents a tail index.  If the first index is not preceded by anything, it is assumed to be a normal index.

There need not be any indices.  @ is a valid dt_ref.

binary dtype representation
***************************

This section details the binary representation of each of the dtypes.

int:		[32-bit signed integer in network byte order]
unsigned:	[32-bit unsigned integer in network byte order]

dt_null: 	0x00000001

dt_int:		0x00000002 [int]

dt_string:	0x00000008 [unsigned length (includes trailing '\0')] ['length' bytes of string data, with the last byte '\0']

dt_bang:	0x00000010

dt_vector:	0x00000040 [unsigned length] [list of 'length' dtypes]

dt_ref:		[don't need]

dt_real:	[don't need]

dt_packet:	[don't need]

Environments, evaluator, and servers
************************************

There are three major applications for dtypes which are explicitly supported by the dtype library.  These are dtype Environments, dtype Evaluator, and dtype Servers.

dtype Environments
==================

dtypes are an especially convenient way to store information expressed as keyword/value pairs.  For example, a personal profile might be stored in such a manner:

     (
      ("name" "john")
      ("address" (38 "Hopeless Ave."))
      ("idnumber" 367552)
      ("interests" ("hockey" "music" "farming"))
     )

In this case, the dtype is a dt_vector of two element dt_vectors, where the first element of each of those lists is a string.  The overall dtype is called an environment, and the two element dt_vectors are called bindings.  The first element of a binding is called a keyword, the second element of the binding is called a value.  For example, one of the bindings is ("address" (38 "Hopeless Ave.")).  In this case, the keyword is "address", and the value is (38 "Hopeless Ave.").  The keyword in a binding need not be a string, although this will usually be the case.

Technically, an environment can be of three different forms.  A single binding is considered an environment.  A list of bindings is also considered an environment, and is the most common form of an environment.  The third form of an environment is to make a list of "normal" environments.  This allows entire lists of bindings to be added or removed from the environment.

dtype environment methods
-------------------------

The dtype library supports dtype environments by providing search routines for use with dtype environments.  Assuming DT is a dt_vector * holding a dtype environment:

dt->env_lookup(const char *pkeyword)
     Searches the environment dt for the first a binding whose keyword is equivalent to the string pkeyword.  If found, then a pointer to the value portion of that binding is returned.  Using the above environment as an example, dt.env_lookup("idnumber") would return a pointer to the dtype in the environment with value 367552.  The bindings are searched in the order that they appear in the environment.  If the binding is not found, then NULL is returned.

dt->env_lookup_binding(const char *pkeyword)
     Does the same thing as env_lookup, except that a pointer to the entire binding is returned, rather than just the value.  For the above example, dt.env_lookup("idnumber") would return ("idnumber" 367552).

dt->env_lookup(dtype *pkeydt)
     Same as dt.env_lookup for string keywords, except that this will search for other dtype types as the keyword.

dt->env_lookup_binding(dtype *pkeydt)
     Same as dt.env_lookup_binding for string keywords, except that this will search for other dtype types as the keyword.

Routines are also provided to make it easy to add and remove bindings from an environment.  These routines should be used, rather than going in and doing the list manipulations yourself.

dt->envAdd(const char *pkeyword, dtype *pvalue)
     Adds a new binding to the environment with the given pkeyword and pvalue.  This is functionally equivalent to:

          (*dt) += new dt_vector(*new dt_string(pkeyword), pvalue, NULL);

     Returns a pointer to the new binding.

dt->env_set(const char *pkeyword, dtype *pvalue, [int pfree])
     "Sets" a value in the environment, by either modifying an existing binding or adding a new binding.  If pkeyword already has a binding in the environment, then the value portion of that binding is replaced with pvalue.  If pfree is set to DT_FREE, then the old value is freed.  If pkeyword does not already have a binding in the environment, then a new one is created.  The binding that was found or created is returned.

dt->env_remove(const char *pkeyword)
     Removes and destroys the first binding in the environment with the given pkeyword.  Note that the entire binding is removed and destroyed, not just the value.  Returns 0 if successful, -1 if the binding was not found.

dtype Environment References
----------------------------

The dtype library also supports the notion of dtype Environment References.  This mechanism allows references to an environment to be embedded within a dtype.  When a dtype containing dtype Environment References is dereferenced through an environment, all those Environment References are looked up in the environment and replaced with their values.

A dtype Environment Reference has the following form:

     (<dt_ref> <dtype keyword>)

To dereference this Environment Reference, the dtype keyword is looked up in the environment, the dt_ref is applied to the value found there, and the result replaces the Environment Reference.

Dereferencing a dtype involves recursively traversing the dtype (if it is a dt_vector) and replacing all Environment References with their values.  For example, consider the following dtype:

     (1 2 3 (@ "name") (4 (3 (@1 "interests")) (@,1 "interests")))

After dereferencing this dtype through the example environment, the dtype will be modified to look like this:

     (1 2 3 "john" (4 (3 "music") ("music" "farming")))

When the dtype Environment References are replaced by their looked-up values, the Environment References are automatically freed.  That is not the programmers responsibility.

The dtype library has one routine for dereferencing a dtype through an environment:

dt->env_dereference(dtype *dtenv)
     Dereferences dt through the dtype environment dtenv.  A pointer to the modified dt is returned.

dtype evaluator
===============

dtypes can represent programming statements.  The dtype library provides support for a programming language that uses dtypes to represent programs.  This programming language is called DEVAL, short for dtype EVALuator, and is an extremely watered-down version of LISP or SCHEME.

The reason for using dtypes to store programs is that these programs can then benefit from all the functionality available to dtypes.  They can be transmitted across the network, they can be dynamically loaded into a running process, they are portable across all machines, etc. The disadvantages to using DEVAL is that it is somewhat clunky to program in, and it is also an interpreted language.

However, the biggest appeal for DEVAL is that it is easily expandable. A programmer can quickly add to the list of functions in the DEVAL language.  Those functions can also be local to a single application - they don't have to become built-in features of the DEVAL language.

A special class, called a dt_eval, is used to evaluate DEVAL programs.  This class has a single constructor:

dt_eval dte

Once a dt_eval has been constructed, it can be used to evaluate any number of DEVAL programs, using the following method:

dte.eval(dtype *pdt, dtype *penv)
     Evaluates DEVAL expression pdt through environment penv.  The result is returned as a dtype *.

The DEVAL language
------------------

A DEVAL expression is a single dtype, and every dtype is a DEVAL expression.  If this dtype is a dt_vector, then it might contain dtypes within it, which are themselves DEVAL expressions.  A program in DEVAL is a single DEVAL expression.

A DEVAL program is run by evaluating a DEVAL expression. Evaluating an expression always results in a new dtype, and may also produce side-effects, such as printing or changing the value of a variable.

Expressions are always evaluated in the context of a dtype environment, which serves as the variable space for the program.  As an expression is evaluated, it can add variables to the environment or look up the values of variables in the environment.

The rules for evaluating a dtype depend on the type of the dtype.  The rules are as follows:

dt_null, dt_bang, dt_int, dt_real, dt_ref, dt_packet
     These dtypes all evaluate to themselves.  Specifically, they evaluate to a deep copy (copyR) of themselves.

dt_string
     A dt_string dtype is considered a variable reference.  The dt_string is evaluated by looking up the dt_string as a keyword to search the dtype environment.  The result of the evaluation is the value that is found in that search.  If the variable is not found, then an error message is printed out and dt_null is the result of the evaluation.

dt_vector
     A dt_vector can be evaluated one of two ways, depending on what the first element of the list is:

     dt_ref
          If the first element of the list is a dt_ref, then the list must have two elements.  The second element of the list is evaluated, the dt_ref is applied to the second element, and the result is the result of the evaluation.

     dt_string
          This is the equivalent of a function call, where the dt_string is the name of the function, and the remaining elements of the list are the arguments to the function call.  The function will perform some operation based on the arguments passed to it, then come up with a dtype as its result, which is the result of the evaluation.  The function might also generate side-effects, such as printing or changing a variable.  In general, a function will evaluate all of its arguments, but this need not be the case (the if function is a canonical example).

     If the dt_vector falls into neither of the above two categories, then an error message is printed and dt_null is the result of the evaluation.

That's all there is to the operation of the DEVAL language.  All the power of the language is contained in its functions.

There are two major simplifications which makes DEVAL a bare-bones language.  The first is that there is only a single, global, variable space.  There is no notion of local versus global variable spaces.  The second simplification is that there is no explicit provision for a DEVAL program to define its own functions.  All functions called in a DEVAL program are functions which have been written in C and are built into the DEVAL language, or have been added to the language by a specific application.

Built-in DEVAL language functions
---------------------------------

The following functions are built into the DEVAL language and come with the dtype library:

How to add your own DEVAL functions
-----------------------------------

New functions can easily be added to the DEVAL language.  These functions do not necessarily have to be added to the dtype library - they can be added by a specific application, for that specific application.

A DEVAL function is implemented as a C function.  This C function is passed the function name and a list of the unevaluated arguments.  The C function is expected to return a dtype as its return value.

The dtype evaluator maintains a table of mappings between function names and C functions.  To add new functions to the language, simply write a C function to perform the function, give the function a name, and add the name/C function pointer pair to the dt_eval table using the following method (assuming DTE is a dt_eval *):

dte->add(const char *pfncname, dt_evalFnc *pfncptr, [void *gval], [void *lval]);

     Adds a new function to the table in dte.  The name of the function is pfncname, and pfncptr is a pointer to the function (usually you can just put the C function name here).  Gval and lval are optional additional variables that can be used to pass additional information to the C function --- how those variables are treated is entirely up to the C function.  None of the built-in functions use these variables, but applications which make their own DEVAL functions may wish to use these variables to pass in a pointer to some sort of global information structure.  In general, the GVAL pointer is for global information, used for applications which keep all of their global information in a single structure.  The LVAL pointer is for "local" information, which can be just about anything.

There is a strict policy of memory usage for DEVAL functions.  The rule is that any dtypes that you evaluate are your responsibility to free, and any dtypes that you return or put into the environment are no longer yours to touch.  Thus, a DEVAL function which evaluates all of its arguments must remember to free the results of those evaluations.  The arguments themselves do not have to be freed --- only the results of evaluating them must be freed.

A DEVAL function written in C must have the following prototype:

     dtype *dt_evalFnc(const char *fncname, dtype *args,
                      dtype *env, dt_eval *dte, void *gval, void *lval);

When the dtype Evaluator calls such a function, the name of the function will be passed in through FNCNAME (since several functions might use the same C routine).  A list of the unevaluated arguments is passed in through ARGS, and the dt_eval being used to evaluate the function is passed in through DTE.  GVAL and LVAL pass in the GVAL and LVAL that were specified when the function was added to the dt_eval.  Once the C function is called, it is free to do whatever it wants, as long as it adheres to the memory usage rules above, and returns a pointer to a dtype when it is finished. If the function doesn't know what to return, it should return a new dt_null, not a NULL.

GVAL and LVAL are used to pass in additional values to the C routine, for example, a pointer to a global list of users

Routines to help write new DEVAL functions
..........................................

There are a few "helping" functions which are designed to make it easy to write new DEVAL functions.  These functions do things like checking the types and number of arguments, and evaluating the arguments in the argument list.

int dt_eval::check_num_args(const char *fncname, dtype *args, int min, int max);
     Checks to make sure that the number of arguments in ARGS is between MIN and MAX.  If -1 is supplied for MIN then there is no minimum, and the same is true for MAX.  If everything is ok, then 0 is returned.  Otherwise, an error message is printed out (this is why you must supply FNCNAME) and -1 is returned.

int dt_eval::check_args_all(const char *fncname, dtype *args, int min, int max, int type);
     Same as dt_eval::check_num_args, except that all the arguments are also checked to make sure they are of the specified TYPE.  The different types may be OR'ed together (using |) to allow multiple types.  For example, DT_STRING | DT_INT would allow arguments to be either strings or integers.  If you want to allow all types of dtypes, use DT_ANY.

int dt_eval::check_args(const char *fncname, dtype *args, int min, int max, int type1, int type2, ... , NULL);
     Same as dt_eval::check_args_all, except that the argument types are checked individually.  I.e., the first argument is checked against type1, the second checked against type2, etc.  Don't forget to put the NULL at the end!

dtype *dt_eval::eval_arg_list(dtype *args, dtype *env, dt_eval *dte);
     Evaluates all the arguments in ARGS, using environment ENV and dt_eval DTE.  The result is a new list of all the evaluated arguments, which is returned.  Don't forget, it is your responsibility to free that returned list.

DEVAL function example
......................

The following is an example of how a new DEVAL function is written and added to the language.  Consider the print function, which must have at least one argument.  All the arguments to the print function are evaluated and printed out.  The last argument is returned as the result of the function.

The code for this function looks like this:

     dtype *dt_evalPrint(const char *fncname, dtype *args,
     		   dtype *env, dt_eval *dte)
     {
       dtype *dtargs;
       dtype *ret;

           // Evaluate and check the arguments
       dtargs = dt_eval::eval_arg_list(args, env, dte);
       if(dt_eval::check_args_all(fncname, dtargs, 1, -1, DT_ANY) < 0) {
         delete dtargs;
         return(new dt_null);
       }

           // Print the arguments
       for(int i = 0; i < dtargs->length(); i++) cout << (*dtargs)[i] << endl;

           // Returns the last arguments
       ret = (*dtargs)[-1]->copyR();
       delete dtargs;
       return(ret);
     }

The main thing to note in the code is the memory usage.  The routine evaluates its arguments into DTARGS, so it is the responsibility of the routine to delete dtargs.  However, the routine also returns its last argument, which means that it must not delete that last argument.  Therefore it returns a copyR() of that argument before deleting the argument list.  Also note that DTARGS must be deleted, but ARGS must not be deleted.

The following is an example of code that creates a new dt_eval, adds the print function to the table of functions in dt_eval, and evaluates a dtype expression using that dt_eval.

     dt_eval dte;

       // Add dt_evalPrint to the table of functions
     dte.add("print", dt_evalPrint);

       // Create the dtype expression
     dtype *expr = dtype::parse("(print 3 myname 5)");
       // Create the environment in which to evaluate the expression
     dtype *env = dtype::parse("((myname joe) (age 21))");
       // Evaluate the expression
     dtype *evalexpr = dte.eval(expr, env);
       // Print the result
     cout << "The evaluated expression is: " << evalexpr << endl;

This would produce the following output:

     3
     "joe"
     5
     The evaluated expression is: 5

dtype servers
=============

dtypes are specially suitable for creating network server/client protocols.  dtypes can be communicated over the network in a machine-independent manner, which makes them good carriers of networked information.  The flexibility of dtypes also carries over into a flexible protocol.  A protocol can be expanded while still retaining backwards compatability with older protocols, which means that changing a protocol won't outdate all servers and clients.  Servers and clients which agree to communicate with dtypes have agreed on the form and representation of the data that they will communicate.  This means that they are free to concentrate on agreeing how they will interpret that data, rather than meddling with the mechanics of how that data will be represented and transported.

dtype server/client model
.........................

The dtype server/client model works as follows.  A dtype server runs on a machine, waiting for connections and communications from dtype clients.  dtype clients are free to connect and disconnect themselves from dtype servers as they please.  A dtype server is also capable of handling simultaneous connections from several dtype clients.  When a dtype client wishes to issue a request to the server it is connected to, the dtype client sends a dtype message to the dtype server.  The dtype message is a dt_vector with the following form:

     ("<command>" <arg1> <arg2> ...)

"<COMMAND>" is a dt_string which is the type of request that is being asked of the server.  The rest of the list contains optional arguments for that request.  When the server receives this dtype, it will execute the command, then return a single return dtype to the client.  The format of that return dtype is entirely up to the server.

Every dtype server should document its protocol somewhere.  All this means is that all the available COMMANDS should be listed, along with what ARGUMENTS are expected of those commands, and exactly what RETURN DTYPE can be expected for that command.  In fact, it would be especially nice if every server included the command:

     (help)

which would return a dt_string that lists all the commands available for that server.

Variations on the basic model
.............................

The basic dtype server/client model can handle a wide variety of applications, but often there will need to be slight variations on this model.  Here are a couple of possible variations.

The basic model presumes that the server is essentially a passive entity which only responds to communications from a client.  However, the server can be made to respond to other things as well, such as keyboard input, X events, or a timer timeout.  The timeout is especially important, because it allows the server to initiate its own actions. The general rule is that the server can be made to respond to anything that has a file descriptor, or to a specified timeout.

The way to do this is to slightly modify the main polling loop in the server, specifically the blocking portion of the polling loop.  Instead of just blocking on the network server structure, add blocks for those other things you want to respond to.  Then, in the rest of the polling loop, add in your own checks to see if one of those things you are waiting for actually occurred, and if so take the appropriate action.

Another variation is to move away from the fully synchronous communication protocol where the client must wait for the server's response before it can move on and do something else.  A client is always free to simply send its message to the server and go on with its business without waiting for the server's response.  However, the client must be aware that the server will be sending its response at some time in the future, and that the next communication which the client receives will be that response.  The client must be prepared to recognize that response and either handle it or ignore it as appropriate.

Another variation is to move from the straight server/client model to a server/server model.  In many cases, the differences between a server and a client become less distinct, and eventually it makes more sense to make two servers which have the ability to talk to each other.  There is no problem with this.  A dtype server doesn't care, in fact, doesn't know, if there is a dtype client or server on the other end of its connection.  All that you have to know is how to tell one server to connect to another server.  This can be done with the login() method available to dsys_servers.  See the dsys manual for more information.

Designing and implementing the server
.....................................

A dtype server is essentially a networked version of the DEVAL interpreter.  You should be familiar with the section on the dtype Evaluator Application, and how to write your own new DEVAL functions.

The heart of the dtype server is a single dt_eval.  Whenever a vmessage is received by the server, the message is evaluated by the dt_eval, and the result is returned to the client.  So, putting in your own protocol commands just means adding your own new DEVAL functions to that dt_eval.

The basic elements of server operation are as follows:

First, the server sets up a dsys_bufserver structure to listen for clients.  The server must supply the maximum number of clients the dsys_bufserver will handle, and it must also supply a port number to the dsys_bufserver.  This port number can be 0, indicating that the system should pick a port number, or it should be a service name taken from /etc/services.  Using the service name is preferred, since it means that clients can easily find that server.

Second, the server sets up a dt_eval structure and adds all the DEVAL functions which represent the protocol commands the server understands.

Third, the server runs a polling loop.  This is the main operation of the server, in which the server continually checks all of its clients for new messages.  Whenever a new message is detected, the message is read in and evaluated through the dt_eval.  The return value is then sent back to the client.  Remember to delete the message when you are done with it.

Using a straight polling loop can be a big waste of processor resources, since the server will spend most of its time spinning and waiting for things to happen.  This is why the polling loop includes a blocking statement.  This occurs at the top of the polling loop and essentially says "I don't need any processing cycles until activity is detected on one of these things, or a timeout occurs".  The server will remain in that blocking call, not taking any processing time, until some activity is detected.  At that point the server goes on and executes the rest of the statements in the polling loop, then goes back into the blocking statement.

A dtype server class
....................

The dtype library provides a dt_server class, which handles all the operations of a typical dtype server.  To use this class, the application only has to specify what port number the dt_server should run on, tell the dt_server where the message handlers are, and then poll the dt_server.

dt_server has the following constructors:

dt_server(int PPORT, [const char *PHELPMSG])
     Constructs a dt_server to listen at the specified PPORT number.  If PPORT is 0, then a port number is chosen by the system.  PHELPMSG is an optional argument.  If it is specified, then a handler is automatically added to the server which will respond to the message (help) by sending back the PHELPMSG string to the sender.

dt_server(const char *PSERVNAME, [const char *HELPMSG])
     Same as above, except that a service name is specified in PSERVNAME instead of a port number.

Methods and operators:

dsys_bufserver *getdss()
     Returns a pointer to dsys_bufserver that is being used to communicate with clients.  This allows the program to directly manipulate things within the server.  For example, getdss()->getport() will return the port number that the server is listening on.

dt_eval *getdte()
     Returns a pointer to the dt_eval that is being used to handle messages sent to the server.

void setdebug(int PDEBUG)
     Turns debugging either on or off (if PDEBUG is 0 or 1).  When debugging is on, then messages are printed out whenever a client connects, disconnects, or sends a message.

void add(const char *PFNCNAME, dt_evalFnc *PFNCPTR, [void *gval], [void *lval])
     Adds a message handler to the evaluator in the dt_server. PFNCNAME is the message type that is to be handled.  PFNCPTR is a pointer to the function which will handle that message.  GVAL and LVAL are two additional values that get passed to the message handler whenever it is called.  The form of the message handler is described in the section on the dtype evaluator.

int poll(dsys_pollblock &PDPB)
     Checks the status of the dt_server, allowing it to handle client connections and message.

void setblocks(dsys_pollblock &PDPB)
     Sets all the file descriptors and timeouts required for blocking for the dt_server.  This routine is automatically called by the dsys_pollblock routine addstation().

An example dtype server
.......................

The following is the complete code for a dtype server.  This server keeps "tidbits" of information about people - you can add a new tidbit of information (which is just a dtype) about a person, or you can get one random tidbit about a person, or you can get all the tidbits about a person.  The tidbit database is a dtype environment, where the names are the keyword and the values are lists of the tidbits.  To communicate with the server, use the general dtype client dtypecl.

     /* dtidbitsrv.c

     Creation date - 10/15/92 Nathan Abramson (arista@amt)

     Sample dtype server - holds "tidbits" about people.

     */

     const char *protocol =
     "(newtidbit <name> <dtype>)\n"
     "  Adds a new tidbit for the person specified by name.  Returns 'added'.\n"
     "(gettidbit <name>)\n"
     "  Returns a tidbit about the specified person.\n"
     "(gettidbits <name>)\n"
     "  Returns a list of all the tidbits about the specified person\n"
     "(dumptidbits)\n"
     "  Returns the entire tidbit database\n"
     "(help)\n"
     "  Returns the list of commands\n";

     /*------------------------*/

     #include <stdio.h>
     #include <stdlib.h>
     #include <math.h>
     #include <iostream.h>
     #include "dtype++.h"

     /*------------------------*/
     /* Structure to hold the global state of the server */
     typedef struct GState {
       dtype *db;                   /* Tidbit databse */

       dt_server *dts;               /* The dtype server */
     } GState;

     /*------------------------*/
     /* Message Handlers */

     dtype *HndlrNewtidbit(const char *fncname, dtype *args,
     		       dtype *env, dt_eval *dte, void *gval, void *lval);
     dtype *HndlrGettidbit(const char *fncname, dtype *args,
     		      dtype *env, dt_eval *dte, void *gval, void *lval);
     dtype *HndlrGettidbits(const char *fncname, dtype *args,
     		       dtype *env, dt_eval *dte, void *gval, void *lval);
     dtype *HndlrDumptidbits(const char *fncname, dtype *args,
     			dtype *env, dt_eval *dte, void *gval, void *lval);
     dtype *HndlrNewtidbit(const char *fncname, dtype *args,
     		      dtype *env, dt_eval *dte, void *gval, void *lval);

     /*------------------------*/

     /***********************************************/

     main()
     {
       GState gs;
       int port;
       dsys_pollblock dpb;

           // Set port to 0 so that a port number will be chosen.  For a real
           // service, this port number should be registered as a service in
           // /etc/services, and the service name would be suppled as argument to
           // new dsys_bufserver
       port = 0;
           // Create a server station and figure out its port
       gs.dts = new dt_server(port, protocol);
       cout << "dtidbit server listening on port " <<
         gs.dts->getdss()->getport() << endl;

           // Register the message handlers
       gs.dts->add("newtidbit", HndlrNewtidbit, (void *) &gs, NULL);
       gs.dts->add("gettidbit", HndlrGettidbit, (void *) &gs, NULL);
       gs.dts->add("gettidbits", HndlrGettidbits, (void *) &gs, NULL);
       gs.dts->add("dumptidbits", HndlrDumptidbits, (void *) &gs, NULL);

           // Initialize the database
       gs.db = new dt_vector;

           // MAIN LOOP
       while(1) {
     	// Set up the blocking mechanism to wait for activity on the
     	// server station.  Tack on your own addstation()'s to block
     	// on other dsys stations, or use addread()/write() to block
     	// on your own file descriptors.  For example, adding on
     	// .addread(fileno(stdin)) will drop through the block when
     	// a line of input has been entered into stdin.  Use
     	// .addtimeout() to cause the block to drop through after a
     	// specified time period.
         dpb.clear().addstation(*(gs.dts));
     	// Now block until activity occurs on one of the things we've
     	// set up to block on
         switch(dpb.block()) {
         case -1:
           cerr << "ERROR - Blocking" << endl;
           continue;
         case DSYS_TIMEOUT:
     	  // This will only happen if you have set a timeout on the
     	  // blocker using .addtimeout(), and the specified time period
     	  // has passed by without any other activity.
           cerr << "Timeout" << endl;
           continue;
         }
         dpb.examine();

     	// Poll the server
         gs.dts->poll(dpb);
       }
     }

     /***********************************************/
     /* Message handlers */

     dtype *HndlrNewtidbit(const char *fncname, dtype *args,
     		      dtype *env, dt_eval *dte, void *gval, void *lval)
          /* Handles the "newtidbit" message */
     {
       GState *gs = (GState *) gval;
       dtype *tbits;

           // Check the arguments
       if(dt_eval::check_args(fncname, args, 2, 2,
     		     DT_STRING, DT_ANY, NULL) < 0) {
         return(new dt_string("Bad format"));
       }

           // Find the tidbits for the specified name
       if((tbits = gs->db->env_lookup(args->elem(0)->cval())) == NULL) {
     	// Create a new blank
         tbits = (*(gs->db->envAdd(args->elem(0)->cval(), new dt_vector)))[1];
       }
           // Add a copy of the tidbit to the list
       tbits->append(args->elem(1)->copyR());

           // Make return value
       return(new dt_string("added"));
     }

     dtype *HndlrGettidbit(const char *fncname, dtype *args,
     		      dtype *env, dt_eval *dte, void *gval, void *lval)
          /* Handles the "gettidbit" message */
     {
       GState *gs = (GState *) gval;
       dtype *tbits;
       int tbitssize, tbitnum;

           // Check the arguments
       if(dt_eval::check_args(fncname, args, 1, 1,
     		     DT_STRING, NULL) < 0) {
         return(new dt_string("Bad format"));
       }

           // Find the tidbits for the specified name
       if((tbits = gs->db->env_lookup(args->elem(0)->cval())) == NULL) {
     	// Create a new blank
         tbits = (*(gs->db->envAdd(args->elem(0)->cval(), new dt_vector)))[1];
       }
           // If there's nothing in the tidbits, return NULL.
       if((tbitssize = tbits->length()) == 0) return(new dt_null);
           // Pick one of the tidbits at random and send back a copy
       tbitnum = random() % tbitssize;
       return((*tbits)[tbitnum]->copyR());
     }

     dtype *HndlrGettidbits(const char *fncname, dtype *args,
     		       dtype *env, dt_eval *dte, void *gval, void *lval)
          /* Handles the "gettidbits" message */
     {
       GState *gs = (GState *) gval;
       dtype *tbits;

           // Check the arguments
       if(dt_eval::check_args(fncname, args, 1, 1,
     		     DT_STRING, NULL) < 0) {
         return(new dt_string("Bad format"));
       }

           // Find the tidbits for the specified name
       if((tbits = gs->db->env_lookup(args->elem(0)->cval())) == NULL) {
     	// Create a new blank
         tbits = (*(gs->db->envAdd(args->elem(0)->cval(), new dt_vector)))[1];
       }
       cout << "Tidbits are: " << gs->db << endl;

           // Return a copy of the tidbits
       return(tbits->copyR());
     }

     dtype *HndlrDumptidbits(const char *fncname, dtype *args,
     			dtype *env, dt_eval *dte, void *gval, void *lval)
          /* Handles the "dumptidbits" message */
     {
       GState *gs = (GState *) gval;
       dtype *tbits;

           // Check the arguments
       if(dt_eval::check_args(fncname, args, 0, 0, NULL) < 0) {
         return(new dt_string("Bad format"));
       }

       cout << "Tidbits are: " << gs->db << endl;

           // Return a copy of the entire database
       return(gs->db->copyR());
     }

     dtype *HndlrHelp(const char *fncname, dtype *args,
     		 dtype *env, dt_eval *dte, void *gval, void *lval)
          /* Handles the "help" message */
     {
       GState *gs = (GState *) gval;

           // Check the arguments
       if(dt_eval::check_args(fncname, args, 0, 0, NULL) < 0) {
         return(new dt_string("Bad format"));
       }

           // Return the protocol string
       return(new dt_string(protocol));
     }

     /***********************************************/

Designing and implementing the Client
.....................................

A dtype client is very simple to add into an application.  The only thing you need is to create a dsys_bufclient which is connected to the server you want to talk to.  Then, anytime you want to send a dtype message to the server, send the message using the dtype method write().  Then wait for the server's reply by using the dtype method read().  It's that simple.  Remember to delete your message and the server's reply dtype when you are done with them.

A general dtype Client
......................

The dtype package contains a utility called dtypecl.  This is a general dtype client, which can connect to and communicate with any dtype server.  dtypecl accepts dtypes typed in from stdin, sends them to the dtype server, and prints the server's reply to stdout.

dtypecl can be used to debug servers.  It can also be used as a filter, taking dtypes in from stdin, sending them to the server, and sending the server's replies back through stdout.  Finally, the source code for dtypecl is an excellent example of how to write dtype clients.

The following is the complete source code for dtypecl.

     /* dtypecl.cc

     Creation date - 10/15/92 Nathan Abramson (arista@amt)

     Generalized dtype client, which connects to dtype servers, sends messages
     to the servers, and receives replies.

     */

     #include <stdio.h>
     #include <stdlib.h>
     #include <math.h>
     #include <iostream.h>
     #include <dsys++/dsys++.hh>
     #include <dtype++/dtype++.hh>

          void usage();

     main(int argc, char **argv)
     {
       dtype *dt, *retdt;
       dsys_bufclient dsc;
       int port;
       int ateof;

       /* Make sure the proper number of arguments have been supplied */
       if(argc != 3) {
         usage();
         exit(-1);
       }

           // Connect to the server
       if((port = atoi(argv[2])) == 0) {
     	// A service name was supplied
         if(dsc.connect(argv[1], argv[2]) < 0) {
           cerr << "ERROR - Unable to connect to server" << endl;
           exit(-1);
         }
       }
       else {
     	// A port number was supplied
         if(dsc.connect(argv[1], port) < 0) {
           cerr << "ERROR - Unable to connect to server" << endl;
           exit(-1);
         }
       }

           // Main loop
       while(1) {
         cerr << "Enter dtype to send -> " << endl;
     	// Parse a dtype from tne input stream
         if((dt = dtype::parse(cin, &ateof)) == NULL) {
           if(ateof) break;
           cerr << "ERROR - Unable to parse dtype" << endl;
           continue;
         }
     	// Send the dtype to the server
         if(dt->write(&dsc) < 0) {
           cerr << "ERROR - Unable to send dtype" << endl;
           delete dt;
           continue;
         }
     	// Wait for the reply
         if((retdt = dtype::read(&dsc)) == NULL) {
           cerr << "ERROR - Unable to receive dtype" << endl;
           delete dt;
           continue;
         }
     	// Print the reply
         cerr << "Return dtype -> " << endl;
         cout << retdt << endl;
     	// Free the message and return dtype
         delete dt;
         delete retdt;
       }

           // Close the connection to the server
       dsc.disconnect();
     }

     void usage()
     {
       cerr << "usage: dtypecl <hostname> (<port> or <service>)" << endl;
     }

Index of functions
******************

The following are the shorthands for the various classes: