   This file documents the URL loading package.

   Copyright (C) 1993, 1994, 1995, 1996 William M. Perry Copyright (C)
1996, 1997, 1998, 1999, 2002 Free Software Foundation

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.1 or
any later version published by the Free Software Foundation; with the
Invariant Sections being "GNU GENERAL PUBLIC LICENSE".  A copy of the
license is included in the section entitled "GNU Free Documentation
License."

URL
***

Getting Started
***************

   "Uniform Resource Locators" (URLs) are a specific form of "Uniform
Resource Identifiers" (URI) described in RFC 2396 which updates RFC
1738 and RFC 1808.  RFC 2016 defines uniform resource agents.

   URIs have the form SCHEME:SCHEME-SPECIFIC-PART, where the SCHEMEs
supported by this library are described below.  See Supported URL Types.

   FTP NFS, HTTP, HTTPS, `rlogin', `telnet', tn3270, IRC and gopher
URLs all have the form

     SCHEME://[USERINFO@]HOSTNAME[:PORT][/PATH]

where `[' and `]' delimit optional parts.  USERINFO sometimes takes the
form USERNAME:PASSWORD but you should beware of the security risks of
sending cleartext passwords.  HOSTNAME may be a domain name or a dotted
decimal address.  If the `:PORT' is omitted then the library will use
the `well known' port for that service when accessing URLs.  With the
possible exception of `telnet', it is rare for ports to be specified,
and it is possible using a non-standard port may have undesired
consequences if a different service is listening on that port (e.g. an
HTTP URL specifying the SMTP port can cause mail to be sent).The
meaning of the PATH component depends on the service.

   The library depends on MIME support provided by the `mm-' packages
from Gnus 5.8 or later.  See The MIME library (emacs-mime).

Configuration
=============

 - Variable: url-configuration-directory
     The directory in which URL configuration files, the cache etc.,
     reside.  Default `~/.url'.

Parsed URLs
===========

   The library functions typically operate on "parsed" versions of
URLs.  These are actually vectors of the form:

     [TYPE USER PASSWORD HOST PORT FILE TARGET ATTRIBUTES FULL]

where
TYPE
     is the type of the URL scheme, e.g. `http'

USER
     is the username associated with it, or `nil';

PASSWORD
     is the user password associated with it, or `nil';

HOST
     is the host name associated with it, or `nil';

PORT
     is the port number associated with it, or `nil';

FILE
     is the `file' part of it, or `nil'.  This doesn't necessarily
     actually refer to a file;

TARGET
     is the target part, or `nil';

ATTRIBUTES
     is the attributes associated with it, or `nil';

FULL
     is `t' for a fully-specified URL, with a host part indicated by
     `//' after the scheme part.

   These attributes have accessors named `url-PART', where PART is the
name of one of the elements above, e.g.  `url-host'.  Similarly, there
are setters of the form `url-set-PART'.

   There are functions for parsing and unparsing between the string and
vector forms.

 - Function: url-generic-parse-url url
     Return a parsed version of the string URL.

 - Function: url-recreate-url url
     Recreates a URL string from the parsed URL.

Retrieving URLs
***************

 - Function: url-retrieve-synchronously url
     Retrieve URL synchronously and return a buffer containing the
     data.  URL is either a string or a parsed URL structure.  Return
     NIL if there are no data associated with it (the case for dired,
     info, or mailto URLs that need no further processing).

 - Function: url-retrieve url callback &optional cbargs
     Retrieve URL asynchronously and call CALLBACK with args CBARGS
     when finished.  The callback is called when the object has been
     completely retrieved, with the current buffer containing the
     object and any MIME headers associated with it.  URL is either a
     string or a parsed URL structure.  Returns the buffer URL will
     load into, or NIL if the process has already completed.

Supported URL Types
*******************

`http' and `https'
==================

   The scheme `http' is Hypertext Transfer Protocol.  The library
supports version 1.1, specified in RFC 2616.  (This supersedes 1.0,
defined in RFC 1945) HTTP URLs have the following form, where most of
the parts are optional:
     http://USER:PASSWORDHOST:PORT/PATH?SEARCHPART#FRAGMENT

   The scheme `https' is a secure version of `http', with transmission
via SSL.  It is defined in RFC 2069.  Its default port is 443.  This
scheme depends on SSL support in Emacs via the `ssl.el' library and is
actually implemented by forcing the `ssl' gateway method to be used.
See Gateways in general.

 - User Option: url-honor-refresh-requests
     This controls honouring of HTTP `Refresh' headers by which servers
     can direct clients to reload documents from the same URL or a or
     different one.  `nil' means they will not be honoured, `t' (the
     default) means they will always be honoured, and otherwise the
     user will be asked on each request.

Cookies
-------

 - User Option: url-cookie-file
     The file in which cookies are stored, defaulting to `cookies' in
     the directory specified by `url-configuration-directory'.

 - User Option: url-cookie-confirmation
     Specifies whether confirmation is require to accept cookies.

 - User Option: url-cookie-multiple-line
     Specifies whether to put all cookies for the server on one line in
     the HTTP request to satisfy broken servers like
     <http://www.hotmail.com>.

 - User Option: url-cookie-trusted-urls
     A list of regular expressions matching URLs from which to accept
     cookies always.

 - User Option: url-cookie-untrusted-urls
     A list of regular expressions matching URLs from which to reject
     cookies always.

 - User Option: url-cookie-save-interval
     The number of seconds between automatic saves of cookies to disk.
     Default is one hour.

Language and Encoding Preferences
---------------------------------

   HTTP allows clients to express preferences for the language and
encoding of documents which servers may honour.

 - User Option: url-mime-charset-string
     This variable specifies a preference for character sets when
     documents can be served in more than one encoding.

     HTTP allows specifying a list of MIME charsets which indicate your
     preferred character set encodings, e.g. Latin-9 or Big5, and these
     can be weighted.  In Emacs 21 this list is generated automatically
     from the list of defined coding systems which have associated MIME
     types.  These are sorted by coding priority.  See Recognizing Coding Systems (emacs).

 - User Option: url-mime-language-string
     A string specifying the preferred language when servers can serve
     files in several languages.  Use RFC 1766 abbreviations, e.g.
     `en' for English, `de' for German.  It can be a comma-separated
     list in descending order of preference.  The ordering can be made
     explicit using `q' factors defined by HTTP, e.g.
     `de, en-gb;q=0.8, en;q=0.7'.  It can be `*' to get the first
     available language (as opposed to the default).

HTTP URL Options
----------------

   HTTP supports an `OPTIONS' method describing things supported by the
URL.

 - Function: url-http-options url
     Returns a property list describing options available for URL.  The
     property list members are:

    `methods'
          A list of symbols specifying what HTTP methods the resource
          supports.

    `dav'
          A list of numbers specifying what DAV protocol/schema
          versions are supported.

    `dasl'
          A list of supported DASL search types supported (string form).

    `ranges'
          A list of the units available for use in partial document
          fetches.

    `p3p'
          The "Platform For Privacy Protection" description for the
          resource.  Currently this is just the raw header contents.

Dealing with HTTP documents
---------------------------

   HTTP URLs are retrieved into a buffer containing the HTTP headers
followed by the body.  Since the headers are quasi-MIME, they may be
processed using the MIME library.  See The MIME library (emacs-mime).
The MIME library doesn't provide a clean function to
do that, so the URL library does.

 - Function: url-decode-text-part handle &optional coding
     This function decodes charset-encoded text in the current buffer.
     In Emacs, the buffer is expected to be unibyte initially and is
     set to multibyte after decoding.  HANDLE is the MIME handle of the
     original part.  CODING is an explicit coding to use, overriding
     what the MIME headers specify.  The coding system used for the
     decoding is returned.

     Note that this function doesn't deal with `http-equiv' charset
     specifications in HTML `<meta>' elements.

file and ftp
============

     ftp://USER:PASSWORD@HOST:PORT/FILE
     file://USER:PASSWORD@HOST:PORT/FILE

   These schemes are defined in RFC 1808.  `ftp:' and `file:' are
synonomous in this library.  They allow reading arbitary files from
hosts.  Either `ange-ftp' (Emacs) or `efs' (XEmacs) is used to retrieve
them from remote hosts.  Local files are accessed directly.

   Compressed files are handled, but support is hard-coded so that
`jka-compr-compression-info-list' and so on have no affect.  Suffixes
recognized are `.z', `.gz', `.Z' and `.bz2'.

 - User Option: url-directory-index-file
     The filename to look for when indexing a directory, default
     `"index.html"'.  If this file exists, and is readable, then it
     will be viewed instead of using `dired' to view the directory.

info
====

     info:FILE#NODE

   Info URLs are not officially defined.  They invoke `Info-goto-node'
with argument `(FILE)NODE'.  `#NODE' is optional, defaulting to `Top'.

mailto
======

   A mailto URL will send an email message to the address in the URL,
for example `mailto:foo@bar.com' would compose a message to
`foo@bar.com'.

 - User Option: url-mail-command
     The function called whenever url needs to send mail.  This should
     normally be left to default from MAIL-USER-AGENT.  See
     Mail-Composition Methods (emacs).

   An `X-Url-From' header field containing the URL of the document that
contained the mailto URL is added if that URL is known.

   RFC 2368 extends the definition of mailto URLs in RFC 1738.  The
form of a mailto URL is
     `mailto:MAILBOX[?HEADER=CONTENTS[&HEADER=CONTENTS]]'
    where an arbitary number of HEADERs can be added.  If the HEADER is
`body', then CONTENTS is put in the body otherwise a HEADER header
field is created with CONTENTS as its contents.  Note that the URL
library does not consider any headers `dangerous' so you should check
them before sending the message.

Email messages are defined in RFC822.

`news', `nntp' and `snews'
==========================

   The network news URL scheme take the following forms following RFC
1738 except that for compatibility with other clients, host and port
fields may be included in news URLs though they are properly only
allowed for nntp an snews.

`news:NEWSGROUP'
     Retrieves a list of messages in NEWSGROUP;

`news:MESSAGE-ID'
     Retrieves the message with the given MESSAGE-ID;

`news:*'
     Retrieves a list of all available newsgroups;

`nntp://HOST:PORT/NEWSGROUP'
`nntp://HOST:PORT/MESSAGE-ID'
`nntp://HOST:PORT/*'
     Similar to the `news' versions.

   `:PORT' is optional and defaults to :119.

   `snews' is the same as `nntp' except that the default port is :563.
(It is tunnelled through SSL.)

   An `nntp' URL is the same as a news URL, except that the URL may
specify an article by its number.

 - User Option: url-news-server
     This variable can be used to override the default news server.
     Usually this will be set by the Gnus package, which is used to
     fetch news.  It may be set from the conventional environment
     variable `NNTPSERVER'.

rlogin, telnet and tn3270
=========================

   These URL schemes from RFC 1738 for logon via a terminal emulator
have the form
     telnet://USER:PASSWORD@HOST:PORT
   but the `:PASSWORD' component is ignored.

   To handle rlogin, telnet and tn3270 URLs, a `rlogin', `telnet' or
`tn3270' (the program names and arguments are hardcoded) session is run
in a `terminal-emulator' buffer.  Well-known ports are used if the URL
does not specify a port.

irc
===

   "Internet Relay Chat" (IRC) is handled by handing off the IRC
session to a function named in `url-irc-function'.

 - User Option: url-irc-function
     A function to actually open an IRC connection.  This function must
     take five arguments, HOST, PORT, CHANNEL, USER and PASSWORD.  The
     CHANNEL argument specifies the channel to join immediately, this
     can be `nil'.  By default this is `url-irc-zenirc'.

 - Function: url-irc-zenirc host port channel user password
     Processes the arguments and lets `zenirc' handle the session.

data
====

     data:[MEDIA-TYPE][;BASE64],DATA

   Data URLs contain MIME data in the URL itself.  They are defined in
RFC 2397.

   MEDIA-TYPE is a MIME `Content-Type' string, possibly including
parameters.  It defaults to `text/plain;charset=US-ASCII'.  The
`text/plain' can be omitted but the charset parameter supplied.  If
`;base64' is present, the DATA are base64-encoded.

nfs
===

     nfs://USER:PASSWORD@HOST:PORT/FILE

   The `nfs:' scheme is defined in RFC 2224.  It is similar to `ftp:'
except that it points to a file on a remote host that is handled by the
automounter on the local host.

 - Variable: url-nfs-automounter-directory-spec
   A string saying how to invoke the NFS automounter.  Certain `%'
sequences are recognized:

`%h'
     The hostname of the NFS server;

`%n'
     The port number of the NFS server;

`%u'
     The username to use to authenticate;

`%p'
     The password to use to authenticate;

`%f'
     The filename on the remote server;

`%%'
     A literal `%'.

   Each can be used any number of times.

cid
===

   RFC 2111

about
=====

ldap
====

   The LDAP scheme is defined in RFC 2255.

imap
====

   RFC 2192

man
===

     `man:PAGE-SPEC'

   This is a non-standard scheme.  PAGE-SPEC is passed directly to the
Lisp `man' function.

Defining New URLs
*****************

Naming conventions
==================

Required functions
==================

Optional functions
==================

Asynchronous fetching
=====================

Supporting file-name-handlers
=============================

General Facilities
******************

Disk Caching
============

   The disk cache stores retrieved documents locally, whence they can be
retrieved mode quickly.  When requesting a URL that is in the cache,
the library checks to see if the page has changed since it was last
retrieved from the remote machine.  If not, the local copy is used,
saving the transmission over the network.  Currently the cache isn't
cleared automatically.

 - User Option: url-automatic-caching
     Setting this variable non-`nil' causes documents to be cached
     automatically.

 - User Option: url-cache-directory
     This variable specifies the directory to store the cache files.
     It defaults to sub-directory `cache' of
     `url-configuration-directory'.

 - User Option: url-cache-creation-function
     The cache relies on a scheme for mapping URLs to files in the
     cache.  This variable names a function which sets the type of
     cache to use.  It takes a URL as argument and returns the absolute
     file name of the corresponding cache file.  The two supplied
     possibilities are `url-cache-create-filename-using-md5' and
     `url-cache-create-filename-human-readable'.

 - Function: url-cache-create-filename-using-md5 url
     Creates a cache file name from URL using MD5 hashing.  This is
     creates entries with very few cache collisions and is fast if you
     have the `md5' function as a primitive (Emacs 21 and XEmacs).
          (url-cache-create-filename-using-md5 "http://www.example.com/foo/bar")
            => "/home/fx/.url/cache/fx/http/com/example/www/b8a35774ad20db71c7c3409a5410e74f"

 - Function: url-cache-create-filename-human-readable url
     Creates a cache file name from URL more obviously connected to URL
     than for `url-cache-create-filename-using-md5', but more likely to
     conflict with other files.
          (url-cache-create-filename-human-readable "http://www.example.com/foo/bar")
            => "/home/fx/.url/cache/fx/http/com/example/www/foo/bar"

Proxies and Gatewaying
======================

   Proxy servers are commonly used to provide gateways through firewalls
or as caches serving some more-or-less local network.  Each protocol
(HTTP, FTP, etc.) can have a different gateway server.  Proxying is
conventionally configured commonly amongst different programs through
environment variables of the form `PROTOCOL_proxy', where PROTOCOL is
one of the supported network protocols (`http', `ftp' etc.).  The
library recognizes such variables in either upper or lower case.  Their
values are of one of the forms:
   * `HOST:PORT'

   * A full URL;

   * Simply a host name.

   The `NO_PROXY' environment variable specifies URLs that should be
excluded from proxying (on servers that should be contacted directly).
This should be a comma-separated list of hostnames, domain names, or a
mixture of both.  Asterisks can be used as wildcards, but other clients
may not support that.  Domain names may be indicated by a leading dot.
For example:
     NO_PROXY="*.aventail.com,home.com,.seanet.com"
    says to contact all machines in the `aventail.com' and `seanet.com'
domains directly, as well as the machine named `home.com'.  If
`NO_PROXY' isn't defined, `no_PROXY' and `no_proxy' are also tried, in
that order.

Proxies may also be specified directly in Lisp.

 - User Option: url-proxy-services
     This variable is an alist of URL schemes and proxy servers that
     gateway them.  The items are of the form
     `(SCHEME . HOST:PORTNUMBER)', says that the URL SCHEME is
     gatewayed through PORTNUMBER on the specified HOST.  An exception
     is the pseudo scheme `"no_proxy"', which is paired with a regexp
     matching host names not to be proxied.  This variable is
     initialized from the environment as above.

          (setq url-proxy-services
                '(("http"     . "proxy.aventail.com:80")
                  ("no_proxy" . "^.*\\(aventail\\|seanet\\)\\.com")))

Gateways in General
===================

   The library provides a general gateway layer through which all
networking passes.  It can both control access to the network and
provide access through gateways in firewalls.  This may make direct
connexions in some cases and pass through some sort of gateway in
others.(1)  The library's basic function responsible for making
connexions is `url-open-stream'.

 - Function: url-open-stream name buffer host service
     Open a stream to HOST, possibly via a gateway.  The other
     arguments are as for `open-network-stream'.  This will not make a
     connexion if `url-gateway-unplugged' is non-`nil'.

 - Variable: url-gateway-local-host-regexp
     This is a regular expression that matches local hosts that do not
     require the use of a gateway.  If `nil', all connexions are made
     through the gateway.

 - Variable: url-gateway-method
     This variable controls which gateway method is used.  It may be
     useful to bind it temporarily in some applications.  It has values
     taken from a list of symbols.  Possible values are:

    `telnet'
          Use this method if you must first telnet and log into a
          gateway host, and then run telnet from that host to connect
          to outside machines.

    `rlogin'
          This method is identical to `telnet', but uses `rlogin' to
          log into the remote machine without having to send the
          username and password over the wire every time.

    `socks'
          Use if the firewall has a SOCKS gateway running on it.  The
          SOCKS v5 protocol is defined in RFC 1928.

    `native'
          This method uses Emacs's builtin networking directly.  This
          is the default.  It can be used only if there is no firewall
          blocking access.

   The following variables control the gateway methods.

 - User Option: url-gateway-telnet-host
     The gateway host to telnet to.  Once logged in there, you then
     telnet out to the hosts you want to connect to.

 - User Option: url-gateway-telnet-parameters
     This should be a list of parameters to pass to the `telnet'
     program.

 - User Option: url-gateway-telnet-password-prompt
     This is a regular expression that matches the password prompt when
     logging in.

 - User Option: url-gateway-telnet-login-prompt
     This is a regular expression that matches the username prompt when
     logging in.

 - User Option: url-gateway-telnet-user-name
     The username to log in with.

 - User Option: url-gateway-telnet-password
     The password to send when logging in.

 - User Option: url-gateway-prompt-pattern
     This is a regular expression that matches the shell prompt.

 - User Option: url-gateway-rlogin-host
     Host to `rlogin' to before telnetting out.

 - User Option: url-gateway-rlogin-parameters
     Parametres to pass to `rsh'.

 - User Option: url-gateway-rlogin-user-name
     User name to use when logging in to the gateway.

 - User Option: url-gateway-prompt-pattern
     This is a regular expression that matches the shell prompt.

 - User Option: socks-server
     This specifies the default server, it takes the form
     `("Default server" SERVER PORT VERSION)' where VERSION can be
     either 4 or 5.

 - Variable: socks-password
     If this is `nil' then you will be asked for the passward,
     otherwise it will be used as the password for authenticating you to
     the SOCKS server.

 - Variable: socks-username
     This is the username to use when authenticating yourself to the
     SOCKS server.  By default this is your login name.

 - Variable: socks-timeout
     This controls how long, in seconds, to wait for responses from the
     SOCKS server; it is 5 by default.

 - User Option: socks-nslookup-program
     This the `nslookup' program.  It is `"nslookup"' by default.

   ---------- Footnotes ----------

   (1) Proxies (which only operate over HTTP) are implemented using
this.

Suppressing Network Connexions
------------------------------

   In some circumstances it is desirable to suppress making network
connexions.  A typical case is when rendering HTML in a mail user
agent, when external URLs should not be activated, particularly to
avoid `bugs' which `call home' by fetch single-pixel images and the
like.  To arrange this, bind the following variable for the duration of
such processing.

 - Variable: url-gateway-unplugged
     If this variable is non-`nil' new network connexions are never
     opened by the URL library.

Broken Hostname Resolution
--------------------------

   Some C libraries do not include the hostname resolver routines in
their static libraries.  If Emacs was linked statically, and was not
linked with the resolver libraries, it wil not be able to get to any
machines off the local network.  This is characterized by being able to
reach someplace with a raw ip number, but not its hostname
(<http://129.79.254.191/> works, but <http://www.cs.indiana.edu/>
doesn't).  This used to happen on SunOS4 and Ultrix, but is now
probably now rare.  If Emacs can't be rebuilt linked against the
resolver library, it can use the external `nslookup' program instead.

 - User Option: url-gateway-broken-resolution
     If non-`nil', this variable says to use the program specified by
     `url-gateway-nslookup-program' program to do hostname resolution.

 - User Option: url-gateway-nslookup-program
     The name of the program to do hostname lookup if Emacs can't do it
     directly.  This program should expect a single argument on the
     command line--the hostname to resolve--and should produce output
     similar to the standard Unix `nslookup' program:
          Name: www.cs.indiana.edu
          Address: 129.79.254.191

History
=======

   The library can maintain a global history list tracking URLs
accessed.  URL completion can be done from it.  The history mechanism
is set up automatically via `url-do-setup' when it is configured to be
on.  Note that the size of the history list is currently not limited.

   The history `list' is actually a hash table,
`url-history-hash-table'.  It contains access times keyed by URL
strings.  The times are in the format returned by `current-time'.

 - Function: url-history-update-url url time
     This function updates the hsitory table with an entry for URL
     accessed at the gievn TIME.

 - User Option: url-history-track
     If non-`nil', the library will keep track of all the URLs
     accessed.  If is is `t', the list is saved to disk at the end of
     each Emacs session.  The default is `nil'.

 - User Option: url-history-file
     The file storing the history list between sessions.  It defaults to
     `history' in `url-configuration-directory'.

 - User Option: url-history-save-interval
     The number of seconds between automatic saves of the history list.
     Default is one hour.  Note that if you change this variable
     directly, rather than using Custom, after `url-do-setup' has been
     run, you need to run the function `url-history-setup-save-timer'.

 - Function: url-history-parse-history &optional fname
     Parses the history file FNAME (default `url-history-file') and
     sets up the history list.

 - Function: url-history-save-history &optional fname
     Saves the current history to file FNAME (default
     `url-history-file').

 - Function: url-completion-function string predicate function
     You can use this function to do completion of URLs from the
     history.

Customization
*************

Environment Variables
=====================

   The following environment variables affect the library's operation at
startup.

`TMPDIR'
     If this is defined, URL-TEMPORARY-DIRECTORY is initialized from it.

General User Options
====================

   The following user options, settable with Customize, affect the
general operation of the package.

 - User Option: url-debug
     Specifies the types of debug messages the library which are logged
     to the `*URL-DEBUG*' buffer.  `t' means log all messages.  A
     number means log all messages and show them with `message'.  If
     may also be a list of the types of messages to be logged.

 - User Option: url-personal-mail-address

 - User Option: url-privacy-level

 - User Option: url-uncompressor-alist

 - User Option: url-passwd-entry-func

 - User Option: url-standalone-mode

 - User Option: url-bad-port-list

 - User Option: url-max-password-attempts

 - User Option: url-temporary-directory

 - User Option: url-show-status

 - User Option: url-confirmation-func
     The function to use for asking yes or no functions.  This is
     normally either `y-or-n-p' or `yes-or-no-p', but could be another
     function taking a single argument (the prompt) and returning `t'
     only if an affirmative answer is given.

 - User Option: url-gateway-method
     A symbol specifying the type of gateway support to use fro
     connexions from the local machine.  The supported methods are:

    `telnet'
          Run telnet in a subprocess to connect;

    `rlogin'
          Rlogin to another machine to connect;

    `socks'
          Connect through a socks server;

    `ssl'
          Connect with SSL;

    `native'
          Connect directly.