[NTLUG:Discuss] robots.txt

[MadHat]

On Tue, 2003-03-25 at 20:23, Terry Hancock wrote:
> On Tuesday 25 March 2003 05:04 pm, David Ross wrote:
> > I understand that this has to do with search engines,but 
> what should a proper 
> > robots.txt file contain? and what file permissions should 
> it have (444,400??)
> 
> I don't think file permissions are relevant -- if your web 
> server can serve it, that's all that counts.
> 
> Here's an example of what one looks like -- it's basically 
> a list of things you don't want the spiders to search:
> 

Also, if you really want to protect something, put a password on it. 
Not only will some spiders traverse past robots.txt, some people use it
to find the "good stuff" on your site and more than one person has been
bitten by this.

-- 
MadHat at Unspecific.com
`But I don't want to go among mad people,' Alice remarked.
`Oh, you can't help that,' said the Cat: `we're all mad here...'
   -- Lewis Carroll - _Alice's_Adventures_in_Wonderland_