In short it is a method of guiding robots away from certain areas in a
Web server's URL space, by providing a simple text file on the server.
This is especially handy if you have large archives, CGI scripts with
massive URL subtrees, temporary information, or you simply don't want
to serve robots.
After some recent discussions on the Robots mailing list some changes
were proposed, and a new version of this document is now available
from <http://web.nexor.co.uk/mak/doc/robots/robots.html>. Most of the
robots in operation either use it already, or have promised support
soon.
-- Martijn
__________
Internet: m.koster@nexor.co.uk
X-400: C=GB; A= ; P=Nexor; O=Nexor; S=koster; I=M
X-500: c=GB@o=NEXOR Ltd@cn=Martijn Koster
WWW: http://web.nexor.co.uk/mak/mak.html