Author: Preferred Customer
Company: Dell Computer Corporation

IHTFP: An Improved HyperText Filing Proxy

Debajit Ghosh
Professor Agarwal
REC TR 2pm
March 18, 1996

This paper is being submitted for the Phase II Writing Requirement

IHTFP: An Improved HyperText Filing Proxy

Abstract

This paper discusses a proposed design for a large-scale caching system for World Wide Web (WWW) pages.  After examining the criteria the caching system should achieve, this paper reveals some of the issues and possibilities inherent in such a caching system.  It then presents the decisions made in the new IHTFP WWW caching system and compares IHTFP to other web caches used today.
1. Introduction 
If the dramatic growth of the number of web sites in the world (from 130 to 90,000 sites between June 1993 and January 1996[]) is any indication, web traffic clearly accounts for more and more of the traffic on the Internet.  Since much of that traffic consists of repeated requests for the same pages from different clients, a caching system could reduce the amount of traffic on the net.  Reducing traffic on the network is not only important in order to relieve load on network servers, but also becomes a necessity when traffic is costly.

[] http://www.netgen.com/info/growth.html

At the time of this writing, NEARNET is considering charging the MIT community per byte transmitted across the link to the Internet it provides to MIT.  As one might imagine, this could get very expensive for a community that transfers 40,895,505 kilobytes a day[] from the Internet, so reducing the amount of this traffic becomes a key issue.  As mentioned, caching web pages can reduce the amount of traffic on the net; this can save the MIT community a considerable amount of money.  Obviously, relying on an individual browser’s caching system for a single user would not suffice; these caches are quite volatile and are typically much smaller than necessary to make a significant reduction in the amount of web traffic across a network.  Thus, we developed IHTFP, an Improved HyperText Filing Proxy, to provide such a large-scale caching system.  Basically, IHTFP is a proxy server; all web requests are sent to the specified proxy server, where either a cached version of the requested page is returned or a fresh one is retrieved from the source.  This paper outlines many of the issues considered in the design process, highlights and describes the features of IHTFP and compares IHTFP to several other WWW caching systems in existence today.

[] http://web.mit.edu/afs/net/admin/noc/backbone/quarterly

2. Background
	Before describing how the IHTFP system caches web pages, some background knowledge of the workings of the World Wide Web is necessary. This section will provide this knowledge by explaining how the web works and describing an important part of many web systems known as a proxy server.  We assume that the reader has at least used a web browser and done a minimal amount of exploration of the World Wide Web. 
	When a user clicks on a URL (Uniform Resource Locator, an address of a page on the web) in his or her browser, the browser opens a TCP connection to the URL’s target web server and sends a request for the page (this paper uses “page” to describe not only hypertext documents, or HTML formatted files, but also other data and components, such as inline images; document is reserved for a proper HTML file).  The server then sends back the requested item and some additional information headers.  If the requested page is an HTML file, the browser scans the document for component tags such as “<IMG SRC=…>”, denote that the document contains inline data.  If such tags exist, the browser opens new connections to the component URL’s server and requests those components.  The user can then repeat this process and click on links specified within the retrieved web page.
 Many networks now exist behind “firewalls”, security systems that block incoming and/or outgoing traffic.  For this reason, individual users’ machines cannot send web packets to the rest of the network and browse web pages on the WWW.  These networks usually provide a machine known as a proxy server (that can transmit and receive information through the firewall) that accepts web requests from clients within the network, retrieves the requested web pages, and returns them to the clients.  Proxy servers can also be useful in environments not restricted by a firewall, as this paper will discuss and explain.   
3. Design Criteria
	An ideal web caching system would retrieve a web page from an external server only once, place it in a cache, serve the cached copy transparently to clients requesting that page, and automatically update the cached copy whenever the original has been modified.  Of course, no web caching system can flawlessly realize all of these features.  Accordingly, IHTFP prioritizes the features it wishes to achieve from among the ones mentioned above. 
	Specifically, the most important criterion is that IHTFP reduce the amount of web traffic between MIT and NEARNET. The next most important feature is that the system be user-transparent; users should notice no difference between retrieving pages from their source or retrieving them from the cache.   In addition, users should not get stale (outdated) copies of web pages from the cache. Finally, the caching system should be robust, efficient, and as simple as possible while still meeting the other criteria.  
4. Design Issues and Considerations
	IHTFP represents but one of many possible implementations of a caching system that attempts to meet the above criteria.  In fact, it represents a mixture of many ideas and possible implementations of such a system. In order to understand the thought process behind the design of IHTFP, we must first review some of the other possibilities and compare them to the mechanisms present and implemented in IHTFP.  
i) Proxy vs. Transparent:
Most global web caching systems (as opposed to the personal cache that each browser maintains for a specific user) implement the caching system through a proxy server within the client’s network. Because of the popularity of firewalls in networks, proxy servers are already popular, and their very nature make them an obvious place to implement a cache; the cache system could simply intercept web requests and return cached results rather than actually forwarding the request to the content provider.   After much consideration, we decided to implement our web caching system on a proxy server as well; though, we did consider the alternatives and ramifications of our decision, as shown below.
While a proxy-based cache works quite well for networks behind firewalls that must use a proxy server regardless of whether or not it caches responses, it may not be optimal for an open network system of MIT’s scale where user transparency is concerned. In order for users to take advantage of the cache-enabled proxy server, each user must configure his or her browser to specify a proxy server or set one in his or her environment variables.  While reasonable for a relatively small base of users, for a large base, this may not be easy to coordinate. Ideally, in order to be completely transparent, web requests should be automatically be detected and redirected to a cache-enabled proxy. 
Though ideal for providing a completely transparent system, automatic detection of web packets is rather difficult to accomplish.  Some possible mechanisms might be to have some gateway or hub scan each TCP packet’s data for keywords such as HTTP or GET (HTTP specifies that the data is using HTTP, the protocol used on the web; GET is the method in HTTP used to request a page) or determine the destination port of a TCP packet (most web servers use port 80).  However, in order to do this, a gateway or hub must recreate TCP packets from the fragmentation created at the IP layer in order to view the data, and as one might imagine, this process can be time-consuming and expensive.  If the gateway or hub’s “data sniffer” significantly slows down all network requests as they pass through, the resulting system is hardly transparent. 
Thus, IHTFP sacrifices some transparency for network performance.  Additionally, much of the job of providing transparency for end users can be achieved by system administrators.  For example, system administrators could edit the default .cshrc file used by almost all Athena users and set the appropriate environment variable specifying the proxy server.  Plus, staff in charge of the infoagents locker, which houses most of the web browsers supported and used in the Athena environment could change the system wide defaults of the browser to specify the proxy server settings.  Similarly, the Resnet (Residential Networking) staff can make the same changes in the browsers it distributes for the PC and Macintosh platforms, as can the system administrators for UNIX, PC, and Mac labs in various departments and laboratories on the campus.  Thus, only the users that independently acquire and configure browsers (or their dotfiles) need worry about changing browser settings in order to use the proxy server and caching-system.  
ii) Location of the Caching System:
	Another major consideration is where the proxy server should be placed.  Many possibilities exist; for the MIT network, the logical possibilities include placing the proxy on the main FDDI backbone and placing a proxy server on each subnet’s gateway or on each hub.  Other possibilities such as placing them at the NEARNET gateway do not make much sense.  A proxy server at the NEARNET gateway would hinder system scalability if other gateways were added; servers at most other locations would also be inefficient as they might require packets to travel too far (slowing down the system) in order to reach the proxy, for packets from the proxy to reach the NEARNET gateway, or a variation of each. This paper will now examine the logical possibilities in terms of the reliability, simplicity,  and usability provided.
Placing proxies on each gateway represents an interesting possibility; each gateway could have its own localized cache system, and cache misses could propagate to each hub’s cache system, where cache misses there could propagate to an MIT-wide cache system, and so on.  This hierarchical system offers the benefit of not placing too much load on any one computer.  Additionally, though the cache would not be consistent across the different subnets (without a lot of coordinatory work), this layout offers a very localized cache system that enables the similar interests within a subnet to reduce the number of cache misses.  In fact, the Harvest cache system[] implements such a hierarchical cache.  However, this has the serious drawback of placing high complexity and high maintenance requirements on the system; since each gateway will depend on its own cache system, each of these cache systems must be independently maintained.  

[]http://excalibur.usc.edu/

Establishing a cache system on each hub is similar to the above possibility, and shares the same benefits and drawbacks.  Again, proxies located at each hub provides a nice, hierarchical caching system; however, the high maintenance required to set up and keep each of the many (though fewer than above) cache servers running prevents this from being pursued as a possible location for the cache system.  
IHTFP uses a simpler, more traditional location; the caching proxy server is placed in a central, convenient location.  The proxy is simply placed within the MITnet FDDI backbone, so web requests travel directly from user to proxy to NEARNET gateway. Requests therefore do not take an unreasonable amount of time to reach the proxy, and since the entire MIT community shares this proxy server, maintaining a consistent cache is simpler than in an environment with caches in differing locations and user bases.  In addition, if the network topology were changed to add another NEARNET gateway, no changes would have to be made nor proxy servers added.
iii) Number and distribution of proxy servers
Web traffic for a community as large as MIT can be large enough to place a severe load on any one machine.  For this reason, unlike most other web-caching systems, IHTFP is designed to work in a distributed environment. Requests meant to be processed by the proxy server are actually directed to a pool of proxy servers within the MITnet FDDI backbone, much like pools of dialup servers, which lessen the load on any individual machine.  
In order to preserve a uniform cache in this pool, the cached data must be shared between the servers.  There are several possibilities for how this sharing can be implemented.  First, the proxies could use a distributed, shared memory.  This feature has the elegance of automatically providing a uniform cache across all proxies and not really differentiating between what each individual has cached; it is most similar to the traditional single proxy server scenario.  However, it has the serious disadvantage of not being very robust.  A large community will depend heavily on this caching system, and if one proxy dirties or destroys the cache, or if the distributed memory fails (due to individual proxy failure or load or the such), the campus is crippled as far as web requests are concerned.   
Alternatively, the proxies could evenly split the caching amongst themselves and share data via RPC communication. Though not quite as elegant in terms of preserving and presenting a uniform cache, this mechanism is more robust and fault-tolerant; the whole caching system will not fail if an individual proxy fails (there is no common memory that can fail and impair the system).  However, if there are n proxy servers in the pool, and the caching load is truly evenly and randomly distributed, there is an chance of a cached page being in another proxy’s cache.  Thus, a large percentage of the time used to look up a cached result may be spent in the RPC communication between servers, asking the other servers if they have a cached page.  This both slows down the caching system and probably places a nearly equal amount of load on all the caching servers as there would be in a single proxy server environment (since there is a high probability that every proxy server will have to look up each request).  
Thus, IHTFP does not use an evenly distributed caching system.  Instead, it uses the structure of the Internet to organize its cache system.  Within the pool of proxy servers, several proxies are designated to handle requests from .com sites, others handle .edu sites, and the rest handle the other (less prevalent) sites such as .org, .mil, and other countries.  This is a reasonable approximation of anticipated hits to each site, since according to a recent survey, roughly 30% of all hosts and 50% of all websites are .com sites, 20% .edu sites, and the rest comprise significantly fewer sites than either of the above two categories[].

[] http://www.nw.com/zone/WWW/dist-bynum.html

When specifying a proxy server within their environment or browser, users actually specify IHTFP’s “master proxy server”.  This master server, aware of all the proxy servers in the pool,  then redirects the request to a .com sub-pool, .edu sub-pool, or another sub-pool of proxies, as appropriate.  Upon redirection to a sub-pool, one specific proxy is chosen at random and attempts to handle the request.  If the chosen proxy’s cache does not contain the requested page, it polls the other proxies in its sub-pool for the page via an RPC call.  If these calls fail to produce a cached page as well, the chosen server retrieves the page from the source.  While this also is subject to being slow and loading the peer proxies within its sub-pool, the reduced number of target machines makes this more manageable and reasonable.  Further, speed in returning pages is not the top priority in IHTFP; reducing the amount of Internet traffic in a reasonably efficient manner is.  IHTFP’s distribution mechanism attempts to accomplish just that. 

Request for pages from an IP address specified server (opposed to a server specified by a hostname) go to the third category.

This could be done by giving each machine in a sub-pool the same name, as with the dialup machines, and rotating to which IP address a name resolves.

iv) Cachable vs. Uncachable data
	Another issue that caching systems, especially web caching systems, must consider is what data can be considered cachable and what cannot. Since usage patterns can differ greatly, IHTFP leaves much of this as configurable; however, many defaults and standards are provided.  The main considerations in determining whether or not data can be cached are whether or not it is volatile and user or time dependent and whether or not data should be considered “private.”  
	Most user and time dependent data, such as searches, are received from CGI scripts (a script or program that is run on the remote web server; usually the URL contains “cgi”,“cgi-bin”, and sometimes contains “?” for denoting program arguments) and fill-out forms (such as comment forms); thus neither type of request should be cached.  Either the master proxy server or individual proxy servers can filter out such requests; leaving this to the individual proxy is recommended in order to reduce the load on the master proxy.  Upon receipt of a response, the proxy server returns the response to the original client without caching the response.  These requests can be filtered by examining the request for keywords such as “cgi” or “?” in the requested URL or the “POST”[] method in the request.   The administrator can also specify other expressions in the URL that deem a request uncachable. A final type of time-dependent data consists of Netscape’s Server-Push connection; basically, a server-push document keeps the connection between the client and web server open and continuously updates the data.  This also should not be cached and can be detected and filtered out by scanning the returned document for the "multipart/x-mixed-replace" MIME header.

[] http://www.w3.org/pub/WWW/Protocols/HTTP/EDIT_ME

The POST method is used to post data from a form to the web server

[] http://www.netscape.com/assist/net_sites/pushpull.html

Multimedia header

	Most “secure” or private data are also received as results of a POST method; most current web servers desiring some form of security or password protection ask for a password in a form.  Thus, the above restriction against caching POST-produced data suffices for the most part. Also of concern are Secure Socket Layer connections between the client and the server, which basically provide secure, encrypted TCP/IP connections between a the requester and web server.  Obviously, this information should not be cached either; IHTFP detects and rejects (for caching) these connections by checking the URL for the https:// protocol specifier, the protocol used by SSL.  

[] http://home.netscape.com/newsref/ref/netscape-security.html#ssl

	Finally, other than GET and POST, the other popular method used in HTTP to retrieve data is the HEAD method.  This method simply returns header information about a page and is most often used to determine whether or not a document has changed since the last check; because of this type of use, IHTFP does not cache responses from the HEAD method.  Thus, since these three methods are the only ones widely used on the web (in fact, some web servers only allow these three methods), rather than scanning and restricting HEAD and POST method requests, IHTFP only allows requests that start with the characters “GET” to be cached.  

[] http://www.w3.org/pub/WWW/Protocols/HTTP/EDIT_ME

	One type of document that should not be cached but usually will be is one that uses a server-side include (a directive that tells the server to include some data in real-time before serving a document, such as inserting a counter or current time into the document).   Unfortunately, the document that the server returns has not traces of the original server-side include, so these documents cannot be distinguished from normal ones and accordingly are cached as normal.  
v) Cache details
	Simpler than conventional cache systems, web caches are primarily a read-only cache, so details such as write-through and snooping are largely irrelevant. No web-client can write or change a cached web page; individual proxies write to the cache only after a web page is retrieved.
	Accordingly, the primary consideration of the cache is the replacement algorithm used.  The most prevalent algorithms in most caching systems are LRU (Least Recently Used), FIFO (first in first out), and random; in a web caching system, a variation of FIFO seems to be the most reasonable.  This is because an important concern of the system is to ensure the data in the cache is not stale, and the most likely candidate for being stale is the data in the cache closest to expiration. IHTFP thus replaces the entries whose documents are closest to expiration (as defined later in this paper). 
	Another consideration of the cache is the cache size and how the cache is organized and implemented in the proxy server.  As for the size of the cache, any large caching system must be mostly contained within disk space, due to the price differential between disk space and memory, so the key factor is the size of the disk space of any individual proxy server.  Since IHTFP is an easily scalable system (just add more proxy servers to the appropriate pool), no individual server needs to have unreasonably large amounts of disk space.  If each web document is roughly 15K and contains ten components, such as inline images, of about 20K each, a web document and all its components require about 200K of disk space; this is, of course, a very rough, conservative estimate produced from some of the large documents and pages present on the Internet.  Thus, in order to store 10,000 documents and their data, a proxy server should possess 2 Gigabytes of storage, a very reasonable amount of disk space with today’s standards and prices.  
IHTFP could use a simple, one level lookup table and store these files in a single directory; however, as one might imagine, this could get somewhat unmanageable as the number of cached pages reaches its target of roughly 100,000 pages (10,000 documents and all of their components).  Thus, like DEC’s caching relay[], IHFTP uses a hierarchical cache system. Though a large index file is stored in a top level directory, the actual web pages are stored in subdirectories determined by hashing the URL of the web page.  Two subdirectories should suffice; each directory should have 36 (26 letters plus 10 numbers) single character subdirectories.  With ten files in each of the lowest level directory, a total of 100,000 files can be stored in a convenient and organized manner.  

[] http://www.src.dec.com/EDIT_ME

  Another consideration relating to the cache is what exactly constitutes an entry in the cache. A naïve concept of entry would simply be a single web page.  A more intelligent choice, however, would entail both the web document and its components. Only rarely does a browser request a document but not the data it contains, such as its in-line images and the such.  Retrieving or replacing parts of a page serves no purpose and often will slow down the caching system, as parts of a page may be cached but others might still need to be retrieved from the source.  As long as components can be found independently of their documents (useful for when images, etc., are shared amongst several pages), grouping page data together for the purposes of storing and replacing upon cache misses is reasonable and efficient.

For example, if images are turned off or if the user is using a text-based web client, such as Lynx.

vi) Validity of Cached Data
	A major concern of any caching system is ensuring that the cached data is not stale; that is, that the copy of the data in the cache does not differ from the original.  A web cache system has several mechanisms at its disposal to help ensure this criteria is satisfied.  These are contained in a page’s “Meta-Headers”; the headers of primary interest are the “Expires” and “Last Modified” header[].  The “Expires”  header, if present, labels when a document should be considered “stale” in a caching system.  Naturally, this is the ideal header; unfortunately, not many documents use this header.  Many do use the Last-Modified header, which designates the last time the document was modified on the server.  While not a guarantee of when a document will expire, the Last-Modified header can give hints as to how to extrapolate when a document might expire.  IHTFP uses this header in such a manner; as with the W3 httpd cache system, the system uses a configurable factor by which the elapsed time between the last modified and current time is multiplied to give an estimate of when the document may expire.   If neither of these headers are present, the system sets the expire time to a configurable default, such as two days.  In addition, the system does not keep a document around in its cache for more than a configurable amount of time (MAX_CACHE_TIME), such as seven days; if the difference between when a page is being cached and its expiration date exceeds this amount, the expiration date is adjusted reflect this maximum.

[] http://www.w3.org./pub/WWW/Protocols/HTTP/EDIT_ME

[] http://www.w3.org/hypertext/WWW/Daemon/User/Config/Caching.html

	In addition to ensuring validity by removing pages associated with documents closest to expiration, IHTFP checks the validity of requested expired and nearly expired (where nearly expired is a configurable percentage, such as 75% between when the data was cached and when it is due to expire).  For the latter, it performs these checks by sending a request for the data along with the “If-modified-since” qualifier.  This qualifier tells the server (if it supports this feature) only to return the data if it has changed since the supplied date; naturally, IHTFP sends the cache date. Using this qualifier has the benefit of greatly reducing the amount of data transmitted across the Internet from the 10K or 20K range to a mere few hundred bytes.  For expired data, IHTFP again sends a request with the “If-modified-since” qualifier; if the data is unchanged, IHTFP requests the page’s Meta-header information (via the HEAD method) and sets a new expiration date with this information. 

[] http://www.w3.org./pub/WWW/Protocols/HTTP/EDIT_ME

	IHTFP can also save time by automatically “validating” expired and nearly expired documents.  Since the primary criteria is to reduce traffic to and from the Internet, IHTFP does not use this feature by default.  But if desired,   IHTFP can perform this validation during off peak hours; this makes the most sense if a network is charged not for every byte transferred, but rather for the average peak hour bandwidth.   During off-peak hours, when there is a reduced amount of load on the servers, each proxy would send “If-modified-since” requests for nearly expired data refreshes the data as appropriate; this has the benefit of saving server-load for these documents by validating them ahead of time.  This trick of using the If-modified-since request minimizes the amount of web traffic to the Internet but also helps assure that cached data has not been modified since it was written to the cache.
vii) Transparent vs. Visible Cache
	Another issue that almost no other cache systems explore is how visible their caches should be.  Usually caches are completely transparent to the user, except for speeding up access to cached data.  Most web caches adhere to this model; however, it is not unreasonable for a web cache to make its cache visible.  IHTFP does exactly that by having the master proxy server also provide a search engine of the cache.  Since one of the most often used types of resources are search engines, implementing a search engine specifically for cached pages can help keep traffic within MIT.  Instead of immediately using Yahoo or Alta-Vista (the most popular indexes and search engines of the WWW), users within MIT can perform title searches within the IHTFP cache and view these pages, all without having the leave the MIT network. 
5. Implementation Details
	This section will outline the exact process by which a web request is handled in the IHTFP system.  First, the client (configured to use the master proxy as its HTTP proxy server) sends a request to the master proxy server.  Based on whether the destination is a .com, a .edu, or another type of site (including straight IP address), the master proxy server forwards requests to a sub-pool of proxies (probably about three servers in each pool).  One proxy server is chosen randomly from this sub-pool and receives this request.  
The chosen proxy then determines whether or not the request is cachable as described above.  If it is not, the proxy opens a connection to the external server, retrieves the data, and immediately returns the response to the client.  Otherwise, the proxy hashes (using an arbitrary hashing function) the URL and checks the top-level index of its cache for the page.  This index contains the URL, title (if the page is an HTML document), two character hash code, and actual file on the server and is organized by hash code; the proxy can save time by initially checking only appropriate section based on the hash code of the requested URL.  If this fails, the proxy then hashes the referring URL (For component data, such as images, the Referer field, used by the dominant browsers such as Netscape, contains the URL of the component’s associated document) and checks the corresponding section of the index as well.  If this check fails, the proxy then scans its entire top level index; in addition, it sends RPC messages to each of the peer proxies in its subpools, asking each to check their own caches for the URL and return a find, if any.  If the data is found in any of these caches, the proxy returns it to the client; if all of these methods fail, however, a cache-miss occurs, and the proxy retrieves the data.  If the data is a document and contains cachable components, it retrieves each of these cachable components (or waits for an appropriate time-out period).  It then stores this data in a hierarchical cache, under unique filenames in the subdirectory determined by the hashing function, replacing the document (and its components) closest to expiration. It also updates both the top level index as well as another index in the subdirectory containing a group identifier (used to tag documents and their components as being together for the purposes of cache replacement), the actual file name in directory, the cache date, and an expiration date, as determined above.  In addition, if the inline data was retrieved from a source other than the type of the current sub-pool (e.g. the document is from a .com site, but an image was retrieved from a .edu site), the server sends a special RPC message to the master proxy server.  The master proxy server then sends a message to the appropriate sub-pool, telling it to add the data (i.e. the image) to its cache. After all the data has been cached, the proxy returns the original response to the client.  Presumably, the client soon requests the components of the document; during this time the system will lose little time because the data is already cached.
As outlined above, the expiration date is calculated and stored along with a cache entry.  Thus, if a request for a nearly expired or expired item arrives, the system updates the version in its cache using the mechanisms described in a previous section.
A key feature of IHTFP is its scalability; if a larger cache is required, more proxy servers can be added to the appropriate sub-pools.  During these upgrades and system changes, the master proxy must be notified of which computers are being added (or removed) from which sub-pools.  After any changes to the system configuration, the master proxy propagates the updated information about the computers in each sub-pool to the relevant sub-pool; for example, if a new computer was added to the .com sub-pool, the .com sub-pool machines would be notified of the changes and made aware of its peer proxy servers (again, in the case of cache-misses, proxies check their peers’ caches).
Finally, the master proxy server also has its own web-server running for title searches.  When a client submits a title search, the master proxy server sends messages to all of the proxy servers in the pools to search their top-level indexes for the desired title.  The responses from the proxy servers are then collected by the master-proxy server and sent back to the client.
Another key feature is the degree of configurability IHTFP provides.  The features and components of IHTFP that can be configured include:
•	default expiration time
•	what percentage constitutes “nearly expired”
•	the ability to turn automatic validation on and off
•	regular expression lists of restricting what requests are deemed cachable
6. Comparison with other caching systems
	IHTFP borrows and shares many ideas and features with other web caching systems. Implementing the cache in a proxy server is one of the easiest to implement, most network-friendly (in terms of slowing down the network) methods; additionally, given the structure and information provided by the HTTP protocol, there are a limited number of ways to address the concern about the expiration and validity of cached data.  However, IHTFP still manages to bring a new insight and ideas to this previously explored field.  This section will highlight these features and differences and will compare IHTFP to other widespread caching systems.
	Some of the other popular or well-designed web caching system in use today include W3C httpd, the Harvest cache and DEC’s web relay; of these, the former two are the most popular and widely used today, with the Harvest accelerator gaining more and more popularity daily. W3C httpd and DEC’s web relay are rather similar; the caching system is a feature of a single proxy server; distribution of the load complicates the system if maintaining a uniform cache amongst the servers is desired.  Harvest explores and implements many of the hierarchical caching system ideas considered and discussed in this paper, such as having cache-misses on an individual proxy propagate to higher-level servers.  IHTFP probably most closely resembles the Harvest cache; however even these two systems contain significant differences. Table 1 provides a comparison between the key distinguishing features of the above proxy-based caching systems; following the table is an explanation of the presented criteria.

Based on scanning httpd server logs for proxy-type specifications on both the author’s web servers and other web servers.

| transparent | reduce traffic | easily scalable | load distribution | grouping of documents | visibility of cache
W3C httpd | √ | √ | | | |
DEC relay | √ | √ | | | |
Harvest | √ | √ | √ | √ | |
IHTFP | √ | √ | √ | √ | √ | √
Table 1: Comparison IHTFP with other popular web caching systems
•	Transparent: all the systems are more or less transparent, in that they are implemented in a proxy server and require only one-time configuration in order to be used.
•	Reduce traffic: regardless of efficiency of caching mechanisms, all of these systems reduce traffic to the outside network; whenever a cache hit does occur, none of these systems will normally send any data to the outside network.  Further, if the network allows it (i.e. no firewall), not everyone needs to go through the proxy server to reduce the overall network traffic to the Internet; simply by having a reasonable portion of the network’s users using the cache, the network will benefit from some reduction in the amount of Internet traffic.
•	Ease of scalability: Because only Harvest and IHTFP are already designed to be distributed, these two can easily be scaled.  Of course, the other systems can be scaled by adding more disk space or memory to the proxy servers; however, this is not always practical or useful, especially in terms of server load. IHTFP may have a slight advantage in this category, since new machines can be added to scale the cache size with minimal reconfiguration (only need to change settings on master proxy server) or thought of placement; Harvest may require reorganization of the hierarchy of the cache system.
•	Distribution of proxy server load: Again, only Harvest and IHTFP inherently distribute the load amongst several servers.  Harvest does this through hierarchy, while IHTFP accomplishes this by a combination of hierarchy (distributing load to .com, .edu, or other sub-pools) and even distribution, which, as explained above, allows for easier scaling.
•	Grouping of documents with data: None of the other systems seem to group all the data (components) of a document with the document in the cache.  This organization seems to make sense and is seen as a benefit, since usually documents and their components are requested together.  
•	Visible Cache: Again, no other system seems to implement this feature.  This potentially can save a lot of traffic, period, to the outside network by providing one of the most popular web services within the MIT network.
7. Summary and Conclusions
	In conclusion, IHTFP represents a robust, scalable caching system that achieves many of the targeted design criteria.  It brings new ideas and features into a continually explored and developing field.  In developing this system, many issues and alternatives were considered; often, as with any system, IHTFP had to sacrifice some elegance, simplicity, or transparency in order to maintain efficiency and usability. With some of these sacrifices, and also due to the newness of some IHTFP’s, this caching system has some tradeoffs and possible negatives; it also contains room for further exploration and development.  Specifically, IHTFP could benefit from examining and addressing the following hard problems:
•	reducing the chance of bottleneck at the master proxy server.  Although the load is distributed among a pool of proxy servers, all requests need to go through the master proxy server; this represents a potential bottleneck and a possible source of problems.

•	addressing largely ignored issues, such as establishing a method to ignore cached entries and force retrieving web pages from the original source. 

The DEC system accomplished this by modifying the messages the web browser sends to web relay; obviously, this is not user transparent or practical for a large user base or commercial, unmodifiable browser.

•	preventing duplicate storage of the same web page served by different servers, the same server with different names (for example, espn.sportszone.com and espnet.sportszone.com map to the same machines)  This could be done via storing pages under their IP address of their source; that approach, of course, increases the complexity of the system and may create additional Internet traffic during name resolution.
•	improving and extending the functionality of the search engine, perhaps adding keyword searches in an efficient and reasonable manner
All in all, however, the end result is a potentially useful and extensible design that will benefit not only MIT but other environment facing similar issues and concerns about web and Internet traffic as the MIT campus.

References