Re: searchable index of the web
stellr@smyrna.cc.vt.edu (Ray Stell)
From: stellr@smyrna.cc.vt.edu (Ray Stell)
Message-id: <9306301959.AA01276@smyrna.cc.vt.edu>
Subject: Re: searchable index of the web
To: mkgray@athena.mit.edu
Date: Wed, 30 Jun 93 15:59:37 EDT
Cc: www-talk@nxoc01.cern.ch
In-reply-to: <9306301905.AA25385@uranus.MIT.EDU>; from "mkgray@athena.mit.edu" at Jun 30, 93 3:05 pm
X-Mailer: ELM [version 2.3 PL4]
>
> I have written a perl script that wanders the WWW collecting URLs, keeping
> tracking of where it's been and new hosts that it finds. Eventually,
> after hacking up the code to return some slightly more useful information
> (currently it just returns URLs), I will produce a searchabe index of this.
> There is a complete list of all the sites it has found at
>
> <a href="http://www.mit.edu:8001/afs/sipb/user/mkgray/ht/comprehensive.html">
> A complete list of sites found by the W4 (World Wide Web Wanderer)
> </a>
>
> I'll announce here when we get this index properly running, however it probably
> won't be until sometime in August, as I am going on vacation. Until then...
>
> Matthew Gray
> mkgray@athena.mit.edu
>
> Visit the SIPB WWW Plexus Server. URL: http://www.mit.edu:8001/
>
This sounds good, better than depending on my memory. Will w4
crack open an log the <Title> or is this directory tree oriented?
If I'm running a server, how will you know to wander to it?
Should there be a web of wanderers (w5?) so w4e in Europe can
feed from w4u in the US, etc? Why is this topic such a quiet one?
======================================================================
Ray Stell stellr@smyrna.cc.vt.edu (703) 231-4109