[NTLUG:Discuss] robots.txt

[Paul Ingendorf]

User-agent: *
Disallow: /

All readable is fine.

You user agent would corespond to whatever bot you would like to block from
indexing whatever location.  Keep in mind this is not an exact thing it is
mearly a  suggestion for the indexing bot and up to them to obide by.  For
further information use something like google and try searching for
robots.txt you will find tons of information reguarding these files.

Here is one that looks fairly comprehensive.

http://www.searchengineworld.com/robots/robots_tutorial.htm


-----Original Message-----
From: discuss-bounces at ntlug.org [mailto:discuss-bounces at ntlug.org]On
Behalf Of David Ross
Sent: Tuesday, March 25, 2003 7:04 PM
To: ntlug
Subject: [NTLUG:Discuss] robots.txt


I understand that this has to do with search engines,but what should a
proper
robots.txt file contain? and what file permissions should it have
(444,400??)
David
--
this e-mail is certified windows free.

_______________________________________________
https://ntlug.org/mailman/listinfo/discuss