Re: searchable index of the web
mkgray@athena.mit.edu
From: mkgray@athena.mit.edu
Message-id: <9306301905.AA25385@uranus.MIT.EDU>
To: joe@athena.mit.edu
Cc: www-talk@nxoc01.cern.ch
Subject: Re: searchable index of the web
In-reply-to: Your message of Wed, 30 Jun 93 01:39:29 -0400.
<9306300539.AA20609@theodore-sturgeon>
Date: Wed, 30 Jun 93 15:05:45 EDT
I have written a perl script that wanders the WWW collecting URLs, keeping
tracking of where it's been and new hosts that it finds. Eventually,
after hacking up the code to return some slightly more useful information
(currently it just returns URLs), I will produce a searchabe index of this.
There is a complete list of all the sites it has found at
<a href="http://www.mit.edu:8001/afs/sipb/user/mkgray/ht/comprehensive.html">
A complete list of sites found by the W4 (World Wide Web Wanderer)
</a>
I'll announce here when we get this index properly running, however it probably
won't be until sometime in August, as I am going on vacation. Until then...
Matthew Gray
mkgray@athena.mit.edu
Visit the SIPB WWW Plexus Server. URL: http://www.mit.edu:8001/