Working With Multibyte Strings in C
***********************************

Guile allows strings to contain characters drawn from a wide variety of languages, including many Asian, Eastern European, and Middle Eastern languages, in a uniform and unrestricted way.  The string representation normally used in C code --- an array of ASCII characters --- is not sufficient for Guile strings, since they may contain characters not present in ASCII.

Instead, Guile uses a very large character set, and encodes each character as a sequence of one or more bytes.  We call this variable-width encoding a multibyte encoding.  Guile uses this single encoding internally for all strings, symbol names, error messages, etc., and performs appropriate conversions upon input and output.

The use of this variable-width encoding is almost invisible to Scheme code.  Strings are still indexed by character number, not by byte offset; string-length still returns the length of a string in characters, not in bytes.  string-ref and string-set! are no longer guaranteed to be constant-time operations, but Guile uses various strategies to reduce the impact of this change.

However, the encoding is visible via Guile's C interface, which gives the user direct access to a string's bytes.  This chapter explains how to work with Guile multibyte text in C code.  Since variable-width encodings are clumsier to work with than simple fixed-width encodings, Guile provides a set of standard macros and functions for manipulating multibyte text to make the job easier.  Furthermore, Guile makes some promises about the encoding which you can use in writing your own text processing code.

While we discuss guaranteed properties of Guile's encoding, and provide functions to operate on its character set, we do not actually specify either the character set or encoding here.  This is because we expect both of them to change in the future: currently, Guile uses the same encoding as GNU Emacs 20.4, but we hope to change Guile (and GNU Emacs as well) to use Unicode and UTF-8, with some extensions.  This will make it more comfortable to use Guile with other systems which use UTF-8, like the GTk user interface toolkit.

Multibyte String Terminology
============================

In the descriptions which follow, we make the following definitions:

byte
     A byte is a number between 0 and 255.  It has no inherent textual interpretation.  So 65 is a byte, not a character.

character
     A character is a unit of text.  It has no inherent numeric value. A and . are characters, not bytes.  (This is different from the C language's definition of character; in this chapter, we will always use a phrase like ``the C language's char type'' when that's what we mean.)

character set
     A character set is an invertible mapping between numbers and a given set of characters.  ASCII is a character set assigning characters to the numbers 0 through 127.  It maps A onto the number 65, and . onto 46.

     Note that a character set maps characters onto numbers, not necessarily onto bytes.  For example, the Unicode character set maps the Greek lower-case alpha character onto the number 945, which is not a byte.

     (This is what Internet standards would call a "coding character set".)

encoding
     An encoding maps numbers onto sequences of bytes.  For example, the UTF-8 encoding, defined in the Unicode Standard, would map the number 945 onto the sequence of bytes 206 177.  When using the ASCII character set, every number assigned also happens to be a byte, so there is an obvious trivial encoding for ASCII in bytes.

     (This is what Internet standards would call a "character encoding scheme".)

Thus, to turn a character into a sequence of bytes, you need a character set to assign a number to that character, and then an encoding to turn that number into a sequence of bytes.

Likewise, to interpret a sequence of bytes as a sequence of characters, you use an encoding to extract a sequence of numbers from the bytes, and then a character set to turn the numbers into characters.

Errors can occur while carrying out either of these processes.  For example, under a particular encoding, a given string of bytes might not correspond to any number.  For example, the byte sequence 128 128 is not a valid encoding of any number under UTF-8.

Having carefully defined our terminology, we will now abuse it.

We will sometimes use the word character to refer to the number assigned to a character by a character set, in contexts where it's obvious we mean a number.

Sometimes there is a close association between a particular encoding and a particular character set.  Thus, we may sometimes refer to the character set and encoding together as an encoding.

Promised Properties of the Guile Multibyte Encoding
===================================================

Internally, Guile a single encoding for all text --- symbols, strings, error messages, etc.  Here we list a number of helpful properties of Guile's encoding.  It is correct to write code which assumes these properties; code which uses these assumptions will be portable to all future versions of Guile, as far as we know.

Every ASCII character is encoded as a single byte from 0 to 127, in the obvious way.  This means that a standard C string containing only ASCII characters is a valid Guile string (except for the terminator; Guile strings store the length explicitly, so they can contain null characters).

The encodings of non-ASCII characters use only bytes between 128 and 255.  That is, when we turn a non-ASCII character into a series of bytes, none of those bytes can ever be mistaken for the encoding of an ASCII character.  This means that you can search a Guile string for an ASCII character using the standard memchr library function.  By extension, you can search for an ASCII substring in a Guile string using a traditional substring search algorithm --- you needn't add special checks to verify encoding boundaries, etc.

No character encoding is a subsequence of any other character encoding.  (This is just a stronger version of the previous promise.) This means that you can search for occurrences of one Guile string within another Guile string just as if they were raw byte strings.  You can use the stock memmem function (provided on GNU systems, at least) for such searches.  If you don't need the ability to represent null characters in your text, you can still use null-termination for strings, and use the traditional string-handling functions like strlen, strstr, and strcat.

You can always determine the full length of a character's encoding from its first byte.  Guile provides the macro scm_mb_len which computes the encoding's length from its first byte.  Given the first rule, you can see that scm_mb_len (B), for any 0 <= B <= 127, returns 1.

Given an arbitrary byte position in a Guile string, you can always find the beginning and end of the character containing that byte without scanning too far in either direction.  This means that, if you are sure a byte sequence is a valid encoding of a character sequence, you can find character boundaries without keeping track of the beginning and ending of the overall string.  This promise relies on the fact that, in addition to storing the string's length explicitly, Guile always either terminates the string's storage with a zero byte, or shares it with another string which is terminated this way.

Functions for Operating on Multibyte Text
=========================================

Guile provides a variety of functions, variables, and types for working with multibyte text.

Basic Multibyte Character Processing
------------------------------------

Here are the essential types and functions for working with Guile text. Guile uses the C type unsigned char * to refer to text encoded with Guile's encoding.

[[Should the really simple macros assume the text is valid, and just crash if it's not?]]

Note that any operation marked here as a ``Libguile Macro'' might evaluate its argument multiple times.

 -- Libguile Type: scm_char_t
     This is a signed integral type large enough to hold any character in Guile's character set.  All character numbers are positive.

 -- Libguile Macro: scm_char_t scm_mb_get (const unsigned char *P)
     Return the character whose encoding starts at P, or -1 if P does not point to a valid character encoding.

 -- Libguile Macro: int scm_mb_put (unsigned char *P, scm_char_t C)
     Place the encoded form of the Guile character C at P, and return its length in bytes.  If C is not the number of a Guile character, return 0.

 -- Libguile Constant: int scm_mb_max_len
     The maximum length of any character's encoding, in bytes.  You may assume this is relatively small --- less than a dozen or so.

 -- Libguile Macro: int scm_mb_len (unsigned char B)
     If B is the first byte of a character's encoding, return the full length of the character's encoding, in bytes.

 -- Libguile Macro: int scm_mb_char_len (scm_char_t C)
     Return the length of the encoding of the character C, in bytes.

 -- Libguile Function: scm_char_t scm_mb_get_func (const unsigned char *P)
 -- Libguile Function: int scm_mb_put_func (unsigned char *P, scm_char_t C)
 -- Libguile Function: int scm_mb_len_func (unsigned char B)
 -- Libguile Function: int scm_mb_char_len_func (scm_char_t C)
     These are functions identical to the corresponding macros.  You can use them in situations where the overhead of a function call is acceptable, and the cleaner semantics of function application are desireable.

Finding Character Encoding Boundaries
-------------------------------------

These are functions for finding the boundaries between characters in multibyte text.

Note that any operation marked here as a ``Libguile Macro'' might evaluate its argument multiple times, unless the definition promises otherwise.

 -- Libguile Macro: int scm_mb_boundary_p (const unsigned char *P)
     Return non-zero iff P points to the start of a character in multibyte text.

     This macro will evaluate its argument only once.

 -- Libguile Function: const unsigned char * scm_mb_floor (const unsigned char *P)
     ``Round'' P to the previous character boundary.  That is, if P points to the middle of the encoding of a Guile character, return a pointer to the first byte of the encoding.  If P points to the start of the encoding of a Guile character, return P unchanged.

 -- libguile Function: const unsigned char * scm_mb_ceiling (const unsigned char *P)
     ``Round'' P to the next character boundary.  That is, if P points to the middle of the encoding of a Guile character, return a pointer to the first byte of the encoding of the next character.  If P points to the start of the encoding of a Guile character, return P unchanged.

Note that it is usually not friendly for functions to silently correct byte offsets that point into the middle of a character's encoding.  Such offsets almost always indicate a programming error, and they should be reported as early as possible.  So, when you write code which operates on multibyte text, you should not use functions like these to ``clean up'' byte offsets which the originator believes to be correct; instead, your code should signal a text:not-char-boundary error as soon as it detects an invalid offset.  See Multibyte Text Processing Errors.

Multibyte String Functions
--------------------------

These functions allow you to operate on multibyte strings: sequences of character encodings.

 -- Libguile Function: int scm_mb_count (const unsigned char *P, int LEN)
     Return the number of Guile characters encoded by the LEN bytes at P.

     If the sequence contains any invalid character encodings, or ends with an incomplete character encoding, signal a text:bad-encoding error.

 -- Libguile Macro: scm_char_t scm_mb_walk (unsigned char **PP)
     Return the character whose encoding starts at *PP, and advance *PP to the start of the next character.  Return -1 if *PP does not point to a valid character encoding.

 -- Libguile Function: const unsigned char * scm_mb_prev (const unsigned char *P)
     If P points to the middle of the encoding of a Guile character, return a pointer to the first byte of the encoding.  If P points to the start of the encoding of a Guile character, return the start of the previous character's encoding.

     This is like scm_mb_floor, but the returned pointer will always be before P.  If you use this function to drive an iteration, it guarantees backward progress.

 -- Libguile Function: const unsigned char * scm_mb_next (const unsigned char *P)
     If P points to the encoding of a Guile character, return a pointer to the first byte of the encoding of the next character.

     This is like scm_mb_ceiling, but the returned pointer will always be after P.  If you use this function to drive an iteration, it guarantees forward progress.

 -- Libguile Function: const unsigned char * scm_mb_index (const unsigned char *P, int LEN, int I)
     Assuming that the LEN bytes starting at P are a concatenation of valid character encodings, return a pointer to the start of the I'th character encoding in the sequence.

     This function scans the sequence from the beginning to find the I'th character, and will generally require time proportional to the distance from P to the returned address.

     If the sequence contains any invalid character encodings, or ends with an incomplete character encoding, signal a text:bad-encoding error.

It is common to process the characters in a string from left to right. However, if you fetch each character using scm_mb_index, each call will scan the text from the beginning, so your loop will require time proportional to at least the square of the length of the text.  To avoid this poor performance, you can use an scm_mb_cache structure and the scm_mb_index_cached macro.

 -- Libguile Type: struct scm_mb_cache
     This structure holds information that allows a string scanning operation to use the results from a previous scan of the string.  It has the following members:

     character
          An index, in characters, into the string.

     byte
          The index, in bytes, of the start of that character.

     In other words, byte is the byte offset of the character'th character of the string.  Note that if byte and character are equal, then all characters before that point must have encodings exactly one byte long, and the string can be indexed normally.

     All elements of a struct scm_mb_cache structure should be initialized to zero before its first use, and whenever the string's text changes.

 -- Libguile Macro: const unsigned char *scm_mb_index_cached (const unsigned char *P, int LEN, int I, struct scm_mb_cache *CACHE)
 -- Libguile Function: const unsigned char *scm_mb_index_cached_func (const unsigned char *P, int LEN, int I, struct scm_mb_cache *CACHE)
     This macro and this function are identical to scm_mb_index, except that they may consult and update *CACHE in order to avoid scanning the string from the beginning.  scm_mb_index_cached is a macro, so it may have less overhead than scm_mb_index_cached_func, but it may evaluate its arguments more than once.

     Using scm_mb_index_cached or scm_mb_index_cached_func, you can scan a string from left to right, or from right to left, in time proportional to the length of the string.  As long as each character fetched is less than some constant distance before or after the previous character fetched with CACHE, each access will require constant time.

Guile also provides functions to convert between an encoded sequence of characters, and an array of scm_char_t objects.

 -- Libguile Function: scm_char_t *scm_mb_multibyte_to_fixed (const unsigned char *P, int LEN, int *RESULT_LEN)
     Convert the variable-width text in the LEN bytes at P to an array of scm_char_t values.  Return a pointer to the array, and set *RESULT_LEN to the number of elements it contains. The returned array is allocated with malloc, and it is the caller's responsibility to free it.

     If the text is not a sequence of valid character encodings, this function will signal a text:bad-encoding error.

 -- Libguile Function: unsigned char *scm_mb_fixed_to_multibyte (const scm_char_t *FIXED, int LEN, int *RESULT_LEN)
     Convert the array of scm_char_t values to a sequence of variable-width character encodings.  Return a pointer to the array of bytes, and set *RESULT_LEN to its length, in bytes.

     The returned byte sequence is terminated with a zero byte, which is not counted in the length returned in *RESULT_LEN.

     The returned byte sequence is allocated with malloc; it is the caller's responsibility to free it.

     If the text is not a sequence of valid character encodings, this function will signal a text:bad-encoding error.

Exchanging Guile Text With the Outside World in C
-------------------------------------------------

[[This is kind of a heavy-weight model, given that one end of the conversion is always going to be the Guile encoding.  Any way to shorten things a bit?]]

Guile provides functions for converting between Guile's internal text representation and encodings popular in the outside world.  These functions are closely modeled after the iconv functions available on some systems.

To convert text between two encodings, you should first call scm_mb_iconv_open to indicate the source and destination encodings; this function returns a context object which records the conversion to perform.

Then, you should call scm_mb_iconv to actually convert the text. This function expects input and output buffers, and a pointer to the context you got from SCM_MB_ICONV_OPEN.  You don't need to pass all your input to scm_mb_iconv at once; you can invoke it on successive blocks of input (as you read it from a file, say), and it will convert as much as it can each time, indicating when you should grow your output buffer.

An encoding may be stateless, or stateful.  In most encodings, a contiguous group of bytes from the sequence completely specifies a particular character; these are stateless encodings. However, some encodings require you to look back an unbounded number of bytes in the stream to assign a meaning to a particular byte sequence; such encodings are stateful.

For example, in the ISO-2022-JP encoding for Japanese text, the byte sequence 27 36 66 indicates that subsequent bytes should be taken in pairs and interpreted as characters from the JIS-0208 character set.  An arbitrary number of byte pairs may follow this sequence.  The byte sequence 27 40 66 indicates that subsequent bytes should be interpreted as ASCII.  In this encoding, you cannot tell whether a given byte is an ASCII character without looking back an arbitrary distance for the most recent escape sequence, so it is a stateful encoding.

In Guile, if a conversion involves a stateful encoding, the context object carries any necessary state.  Thus, you can have many independent conversions to or from stateful encodings taking place simultaneously, as long as each data stream uses its own context object for the conversion.

 -- Libguile Type: struct scm_mb_iconv
     This is the type for context objects, which represent the encodings and current state of an ongoing text conversion.  A struct scm_mb_iconv records the source and destination encodings, and keeps track of any information needed to handle stateful encodings.

 -- Libguile Function: struct scm_mb_iconv * scm_mb_iconv_open (const char *TOCODE, const char *FROMCODE)
     Return a pointer to a new struct scm_mb_iconv context object, ready to convert from the encoding named FROMCODE to the encoding named TOCODE.  For stateful encodings, the context object is in some appropriate initial state, ready for use with the scm_mb_iconv function.

     When you are done using a context object, you may call scm_mb_iconv_close to free it.

     If either TOCODE or FROMCODE is not the name of a known encoding, this function will signal the text:unknown-conversion error, described below.

     Guile supports at least these encodings:

     US-ASCII
          US-ASCII, in the standard one-character-per-byte encoding.

     ISO-8859-1
          The usual character set for Western European languages, in its usual one-character-per-byte encoding.

     Guile
          Guile's current internal multibyte encoding.  The actual encoding this name refers to will change from one version of Guile to the next.  You should use this when converting data between external sources and the encoding used by Guile objects.

          You should not use this as the encoding for data presented to the outside world, for two reasons.  1) Its meaning will change over time, so data written using the guile encoding with one version of Guile might not be readable with the guile encoding in another version of Guile.  2) It currently corresponds to Emacs-Mule, which invented for Emacs's internal use, and was never intended to serve as an exchange medium.

     Emacs-Mule
          This is the variable-length encoding for multi-lingual text by GNU Emacs, at least through version 20.4.  You probably should not use this encoding, as it is designed only for Emacs's internal use.  However, we provide it here because it's trivial to support, and some people probably do have emacs-mule-format files lying around.

     [[Need an encoding which is Emacs-Mule-UCS-4, so people can do the equivalent of multibyte_to_fixed, but with their own buffers.]]

     (At the moment, this list doesn't include any character sets suitable for external use that can actually handle multilingual data; this is unfortunate, as it encourages users to write data in Emacs-Mule format, which nobody but Emacs and Guile understands.  We hope to add support for Unicode in UTF-8 soon, which should solve this problem.)

     Case is not significant in encoding names.

     [[Need a function which reports whether a given encoding is available.]]

     You can define your own conversions; see Implementing Your Own Text Conversions.

 -- Libguile Function: size_t scm_mb_iconv (struct scm_mb_iconv *CONTEXT, const char **INBUF, size_t *INBYTESLEFT, char **OUTBUF, size_t *OUTBYTESLEFT)
     Convert a sequence of characters from one encoding to another.  The argument CONTEXT specifies the encodings to use for the input and output, and carries state for stateful encodings; use scm_mb_iconv_open to create a CONTEXT object for a particular conversion.

     Upon entry to the function, *INBUF should point to the input buffer, and *INBYTESLEFT should hold the number of input bytes present in the buffer; *OUTBUF should point to the output buffer, and *OUTBYTESLEFT should hold the number of bytes available to hold the conversion results in that buffer.

     Upon exit from the function, *INBUF points to the first unconsumed byte of input, and *INBYTESLEFT holds the number of unconsumed input bytes; *OUTBUF points to the byte after the last output byte, and *OUTBYTELEFT holds the number of bytes left unused in the output buffer.

     For stateful encodings, CONTEXT carries encoding state from one call to scm_mb_iconv to the next.  Thus, successive calls to SCM_MB_ICONV which use the same context object can convert a stream of data one chunk at a time.

     If INBUF is zero or *INBUF is zero, then the call is taken as a request to reset the states of the input and the output encodings.  If OUTBUF is non-zero and *OUTBUF is non-zero, then scm_mb_iconv stores a byte sequence in the output buffer to put the output encoding in its initial state.  If the output buffer is not large enough to hold this byte sequence, scm_mb_iconv returns scm_mb_iconv_too_big, and leaves the shift states of CONTEXT's input and output encodings unchanged.

     The scm_mb_iconv function always consumes only complete characters or shift sequences from the input buffer, and the output buffer always contains a sequence of complete characters or escape sequences.

     If the input sequence contains characters which are not expressible in the output encoding, scm_mb_iconv converts it in an implementation-defined way.  It may simply delete the character.

     Some encodings use byte sequences which do not correspond to any textual character.  For example, the escape sequence of a stateful encoding has no textual meaning.  When converting from such an encoding, a call to scm_mb_iconv might consume input but produce no output, since the input sequence might contain only escape sequences.

     Normally, scm_mb_iconv returns the number of input characters it could not convert perfectly to the ouput encoding.  However, it may return one of the scm_mb_iconv_ codes described below, to indicate an error.  All of these codes are negative values.

     If the input sequence contains an invalid character encoding, conversion stops before the invalid input character, and scm_mb_iconv returns the constant value scm_mb_iconv_bad_encoding.

     If the input sequence ends with an incomplete character encoding, scm_mb_iconv will leave it in the input buffer, unconsumed, and return the constant value scm_mb_iconv_incomplete_encoding.  This is not necessarily an error, if you expect to call scm_mb_iconv again with more data which might contain the rest of the encoding fragment.

     If the output buffer does not contain enough room to hold the converted form of the complete input text, scm_mb_iconv converts as much as it can, changes the input and output pointers to reflect the amount of text successfully converted, and then returns scm_mb_iconv_too_big.

Here are the status codes that might be returned by scm_mb_iconv. They are all negative integers.

scm_mb_iconv_too_big
     The conversion needs more room in the output buffer.  Some characters may have been consumed from the input buffer, and some characters may have been placed in the available space in the output buffer.

scm_mb_iconv_bad_encoding
     scm_mb_iconv encountered an invalid character encoding in the input buffer.  Conversion stopped before the invalid character, so there may be some characters consumed from the input buffer, and some converted text in the output buffer.

scm_mb_iconv_incomplete_encoding
     The input buffer ends with an incomplete character encoding.  The incomplete encoding is left in the input buffer, unconsumed.  This is not necessarily an error, if you expect to call scm_mb_iconv again with more data which might contain the rest of the incomplete encoding.

Finally, Guile provides a function for destroying conversion contexts.

 -- Libguile Function: void scm_mb_iconv_close (struct scm_mb_iconv *CONTEXT)
     Deallocate the conversion context object CONTEXT, and all other resources allocated by the call to scm_mb_iconv_open which returned CONTEXT.

Implementing Your Own Text Conversions
--------------------------------------

This section describes the interface for adding your own encoding conversions for use with scm_mb_iconv.  The interface here is borrowed from the GNOME Project's libunicode library.

Guile's scm_mb_iconv function works by converting the input text to a stream of scm_char_t characters, and then converting those characters to the desired output encoding.  This makes it easy for Guile to choose the appropriate conversion back ends for an arbitrary pair of input and output encodings, but it also means that the accuracy and quality of the conversions depends on the fidelity of Guile's internal character set to the source and destination encodings. Since scm_mb_iconv will be used almost exclusively for converting to and from Guile's internal character set, this shouldn't be a problem.

To add support for a particular encoding to Guile, you must provide one function (called the read function) which converts from your encoding to an array of scm_char_t's, and another function (called the write function) to convert from an array of scm_char_t's back into your encoding.  To convert from some encoding A to some other encoding B, Guile pairs up A's read function with B's write function.  Each call to scm_mb_iconv passes text in encoding A through the read function, to produce an array of scm_char_t's, and then passes that array to the write function, to produce text in encoding B.

For stateful encodings, a read or write function can hang its own data structures off the conversion object, and provide its own functions to allocate and destroy them; this allows read and write functions to maintain whatever state they like.

The Guile conversion back end represents each available encoding with a struct scm_mb_encoding object.

 -- Libguile Type: struct scm_mb_encoding
     This data structure describes an encoding.  It has the following members:

     char **names
          An array of strings, giving the various names for this encoding.  The array should be terminated by a zero pointer.  Case is not significant in encoding names.

          The scm_mb_iconv_open function searches the list of registered encodings for an encoding whose names array matches its TOCODE or FROMCODE argument.

     int (*init) (void **COOKIE)
          An initialization function for the encoding's private data. scm_mb_iconv_open will call this function, passing it the address of the cookie for this encoding in this context.  (We explain cookies below.)  There is no way for the init function to tell whether the encoding will be used for reading or writing.

          Note that init receives a pointer to the cookie, not the cookie itself.  Because the type of COOKIE is void **, the C compiler will not check it as carefully as it would other types.

          The init member may be zero, indicating that no initialization is necessary for this encoding.

     int (*destroy) (void **COOKIE)
          A deallocation function for the encoding's private data. scm_mb_iconv_close calls this function, passing it the address of the cookie for this encoding in this context.  The destroy function should free any data the init function allocated.

          Note that destroy receives a pointer to the cookie, not the cookie itself.  Because the type of COOKIE is void **, the C compiler will not check it as carefully as it would other types.

          The destroy member may be zero, indicating that this encoding doesn't need to perform any special action to destroy its local data.

     int (*reset) (void *COOKIE, char **OUTBUF, size_t *OUTBYTESLEFT)
          Put the encoding into its initial shift state.  Guile calls this function whether the encoding is being used for input or output, so this should take appropriate steps for both directions.  If OUTBUF and OUTBYTESLEFT are valid, the reset function should emit an escape sequence to reset the output stream to its initial state; OUTBUF and OUTBYTESLEFT should be handled just as for scm_mb_iconv.

          This function can return an scm_mb_iconv_ error code (see Exchanging Guile Text With the Outside World in C).  If it returns scm_mb_iconv_too_big, then the output buffer's shift state must be left unchanged.

          Note that reset receives the cookie's value itself, not a pointer to the cookie, as the init and destroy functions do.

          The reset member may be zero, indicating that this encoding doesn't use a shift state.

     enum scm_mb_read_result (*read) (void *COOKIE, const char **INBUF,  size_t *INBYTESLEFT, scm_char_t **OUTBUF, size_t *OUTCHARSLEFT)
          Read some bytes and convert into an array of Guile characters.  This is the encoding's read function.

          On entry, there are *INBYTESLEFT bytes of text at *INBUF to be converted, and *OUTCHARSLEFT characters available at *OUTBUF to hold the results.

          On exit, *INBYTESLEFT and *INBUF indicate the input bytes still not consumed.  *OUTCHARSLEFT and *OUTBUF indicate the output buffer space still not filled.  (By exclusion, these indicate which input bytes were consumed, and which output characters were produced.)

          Return one of the enum scm_mb_read_result values, described below.

          Note that read receives the cookie's value itself, not a pointer to the cookie, as the init and destroy functions do.

     enum scm_mb_write_result (*write) (void *COOKIE, scm_char_t **INBUF, size_t *INCHARSLEFT, **OUTBUF, size_t *OUTBYTESLEFT)
          Convert an array of Guile characters to output bytes.  This is the encoding's write function.

          On entry, there are *INCHARSLEFT Guile characters available at *INBUF, and *OUTBYTESLEFT bytes available to store output at *OUTBUF.

          On exit, *INCHARSLEFT and *INBUF indicate the number of Guile characters left unconverted (because there was insufficient room in the output buffer to hold their converted forms), and *OUTBYTESLEFT and *OUTBUF indicate the unused portion of the output buffer.

          Return one of the scm_mb_write_result values, described below.

          Note that write receives the cookie's value itself, not a pointer to the cookie, as the init and destroy functions do.

     struct scm_mb_encoding *next
          This is used by Guile to maintain a linked list of encodings.  It is filled in when you call scm_mb_register_encoding to add your encoding to the list.

Here is the enumerated type for the values an encoding's read function can return:

 -- Libguile Type: enum scm_mb_read_result
     This type represents the result of a call to an encoding's read function.  It has the following values:

     scm_mb_read_ok
          The read function consumed at least one byte of input.

     scm_mb_read_incomplete
          The data present in the input buffer does not contain a complete character encoding.  No input was consumed, and no characters were produced as output.  This is not necessarily an error status, if there is more data to pass through.

     scm_mb_read_error
          The input contains an invalid character encoding.

Here is the enumerated type for the values an encoding's write function can return:

 -- Libguile Type: enum scm_mb_write_result
     This type represents the result of a call to an encoding's write function.  It has the following values:

     scm_mb_write_ok
          The write function was able to convert all the characters in INBUF successfully.

     scm_mb_write_too_big
          The write function filled the output buffer, but there are still characters in INBUF left unconsumed; INBUF and INCHARSLEFT indicate the unconsumed portion of the input buffer.

Conversions to or from stateful encodings need to keep track of each encoding's current state.  Each conversion context contains two void * variables called cookies, one for the input encoding, and one for the output encoding.  These cookies are passed to the encodings' functions, for them to use however they please.  A stateful encoding can use its cookie to hold a pointer to some object which maintains the context's current shift state.  Stateless encodings will probably not use their cookies.

The cookies' lifetime is the same as that of the context object.  When the user calls scm_mb_iconv_close to destroy a context object, scm_mb_iconv_close calls the input and output encodings' destroy functions, passing them their respective cookies, so each encoding can free any data it allocated for that context.

Note that, if a read or write function returns a successful result code like scm_mb_read_ok or scm_mb_write_ok, then the remaining input, together with the output, must together represent the complete input text; the encoding may not store any text temporarily in its cookie.  This is because, if scm_mb_iconv returns a successful result to the user, it is correct for the user to assume that all the consumed input has been converted and placed in the output buffer. There is no ``flush'' operation to push any final results out of the encodings' buffers.

Here is the function you call to register a new encoding with the conversion system:

 -- Libguile Function: void scm_mb_register_encoding (struct scm_mb_encoding *ENCODING)
     Add the encoding described by *ENCODING to the set understood by scm_mb_iconv_open.  Once you have registered your encoding, you can use it by calling scm_mb_iconv_open with one of the names in ENCODING->names.

Multibyte Text Processing Errors
================================

This section describes error conditions which code can signal to indicate problems encountered while processing multibyte text.  In each case, the arguments MESSAGE and ARGS are an error format string and arguments to be substituted into the string, as accepted by the display-error function.

 -- Condition: text:not-char-boundary func message args object offset
     By calling FUNC, the program attempted to access a character at byte offset OFFSET in the Guile object OBJECT, but OFFSET is not the start of a character's encoding in OBJECT.

     Typically, OBJECT is a string or symbol.  If the function signalling the error cannot find the Guile object that contains the text it is inspecting, it should use #f for OBJECT.

 -- Condition: text:bad-encoding func message args object
     By calling FUNC, the program attempted to interpret the text in OBJECT, but OBJECT contains a byte sequence which is not a valid encoding for any character.

 -- Condition: text:not-guile-char func message args number
     By calling FUNC, the program attempted to treat NUMBER as the number of a character in the Guile character set, but NUMBER does not correspond to any character in the Guile character set.

 -- Condition: text:unknown-conversion func message args from to
     By calling FUNC, the program attempted to convert from an encoding named FROM to an encoding named TO, but Guile does not support such a conversion.

 -- Libguile Variable: SCM scm_text_not_char_boundary
 -- Libguile Variable: SCM scm_text_bad_encoding
 -- Libguile Variable: SCM scm_text_not_guile_char
     These variables hold the scheme symbol objects whose names are the condition symbols above.  You can use these when signalling these errors, instead of looking them up yourself.

Why Guile Does Not Use a Fixed-Width Encoding
=============================================

Multibyte encodings are clumsier to work with than encodings which use a fixed number of bytes for every character.  For example, using a fixed-width encoding, we can extract the Ith character of a string in constant time, and we can always substitute the Ith character of a string with any other character without reallocating or copying the string.

However, there are no fixed-width encodings which include the characters we wish to include, and also fit in a reasonable amount of space. Despite the Unicode standard's claims to the contrary, Unicode is not really a fixed-width encoding.  Unicode uses surrogate pairs to represent characters outside the 16-bit range; a surrogate pair must be treated as a single character, but occupies two 16-bit spaces.  As of this writing, there are already plans to assign characters to the surrogate character codes.  Three- and four-byte encodings are too wasteful for a majority of Guile's users, who only need ASCII and a few accented characters.

Another alternative would be to have several different fixed-width string representations, each with a different element size.  For each string, Guile would use the smallest element size capable of accomodating the string's text.  This would allow users of English and the Western European languages to use the traditional memory-efficient encodings.  However, if Guile has N string representations, then users must write N versions of any code which manipulates text directly --- one for each element size.  And if a user wants to operate on two strings simultaneously, and wants to avoid testing the string sizes within the loop, she must make N*N copies of the loop. Most users will simply not bother.  Instead, they will write code which supports only one string size, leaving us back where we started.  By using a single internal representation, Guile makes it easier for users to write multilingual code.

[[What about tagging each string with its encoding? "Every extension must be written to deal with every encoding"]]

Finally, Guile's multibyte encoding is not so bad.  Unlike a two- or four-byte encoding, it is efficient in space for American and European users.  Furthermore, the properties described above mean that many functions can be coded just as they would for a single-byte encoding; see Promised Properties of the Guile Multibyte Encoding.