The World Wide Web was started originally proposed in 1989 and the first implementation appeared in 1990. The Web however, did not gain any widespread popular use until NCSA Mosaic became available in early 1993.
In Spring of 1993, I wrote the Wanderer to systematically traverse the Web and collect sites. I was initially motivated primarily to discover new sites, as the Web was still a relatively small place. As the Web grew rapidly, the focus quickly changed to charting the growth of the Web. This report covers the period from June 1993 to June 1995.
Tools
The primary tool used to collect the data presented here was the World Wide Web Wanderer, the first automated Web agent or "spider". The Wanderer was first functional in spring of 1993 and performed regular traversals of the Web from June 1993 to June 1995.
Other valuable data sources include data from ZONE, a DNS walker and from the Merit Computer Network Backbone Statistics.
Methodology
The Wanderer was run on a monthly basis from June 1993 to June 1995. While the Wanderer certainly did not reach every site, it was run with consistent methodology, hopefully yielding consistent data for the growth of the Web.
Context
The Internet in general has grown quite fast. Any look at the growth
of the Web necessarily examines the growth of the Internet at large.
Doing so, particularly by looking at the data collected by ZONE and the Merit Computer Network, makes
it all the clearer how fast the Web has grown, even compared to the
explosive growth of the Internet in general.
Results
The Web has grown fast. How fast? For the second half of 1993, the
Web had a doubling period of under 3 months, and even today the
doubling period is still nearly 5 months. Additionally, the
percentage of Web sites that are commercial has increased
dramatically. The numbers below show the percentage growth in the
.com domain, which excludes foreign commercial sites (such as the
.co.uk domain, etc.).
The growth of the Web has been remarkable even compared to the Internet at large, as shown by the number of hosts per Web server. In June of 1995, even with the phenomenal growth of the Internet, the number of Web servers soared to a point where one in every 270 machines on the Internet is a Web server.
| Results Summary | |||
|---|---|---|---|
| Month | # of Web sites | % .com sites | Hosts per Web server |
| 6/93 | 130 | 1.5 | 13,000 |
| 12/93 | 623 | 4.6 | 3,475 |
| 6/94 | 2,738 | 13.5 | 1,095 |
| 12/94 | 10,022 | 18.3 | 451 |
| 6/95 | 23,500 | 31.3 | 270 |
| 1/96 | 100,000 | 50.0 | 94 |