[NTLUG:Discuss] robots.txt / also Nimda & CodeRed fighting...

[Darin W. Smith]

I use:

User-agent: *
Disallow: /

in my robots.txt to not allow robots to search my site at all.  I've seen 
several try.  All of them I've seen in my logs have honored robots.txt.  It 
was their requests for the file that first alerted me to such a file.

I do use passwords on anything privy, or anything that is particularly 
bandwidth-consuming that I don't want the whole world to be pulling 
down...like family photo albums, for both reasons.

Family can get in, and I setup another login for guests I want to let in.  
Only those with a "need to know" are allowed in.

Any other attbi (now Comcast) subscribers noticing a huge upswing in the 
number of Nimda and CodeRed infested machines since the comcast deal was 
completed?  Was the upsurge just co-incidental, since (for my stuff, at 
least) everything is still attbi.com (i.e., no real change in 
administration of the network...just change in ownership)?

I've gone from seeing about a dozen different IPs (mostly other attbi.com 
customers) requesting default.ida in a week's time, to about 3 dozen 
different IPs (also, mostly other attbi.com customers).

I've considered using a 'doze box to do a NET SEND xxx.xxx.xxx.xxx "Please 
scan your computer.  You have a virus/worm." to all those IP's.  Think I'd 
get in trouble?  It would reveal my IP to them.  This is assuming that they 
have the messaging service open...which most probably will since they 
haven't bothered to apply any patches or scan for viruses.

Think it would do any good?

Since I have my webserver (Linux box - Mandrake 8.1 but soon migrating to 
something not French and not bankrupt) setup to automatically find and 
block http traffic from those IPs, I'm not all that concerned about my box. 
 I'm more concerned about those people not knowing that they are 
unwittingly spreading junk like this around to other 'doze users.

Couldn't Comcast automatically block this stuff with packet filters at 
their routers?  Couldn't they also pick off the IP address of infected 
machines and notify the owners that they need to run a virus scan?  It 
seems that would be pretty easy to automate.

In the past, I have sent lists of the attbi.com IP's that are this way to 
abuse at attbi.com or abuse at comcast.com, but I don't think they really care.  
I care, partially because that stuff winds up eating lots of bandwidth.

D!

On 26 Mar 2003 16:08:20 -0600, MadHat <madhat at unspecific.com> wrote:

>
> On Tue, 2003-03-25 at 20:23, Terry Hancock wrote:
>> On Tuesday 25 March 2003 05:04 pm, David Ross wrote:
>> > I understand that this has to do with search engines,but what should a 
>> proper > robots.txt file contain? and what file permissions should it 
>> have (444,400??)
>>
>> I don't think file permissions are relevant -- if your web server can 
>> serve it, that's all that counts.
>>
>> Here's an example of what one looks like -- it's basically a list of 
>> things you don't want the spiders to search:
>>
>
> Also, if you really want to protect something, put a password on it. Not 
> only will some spiders traverse past robots.txt, some people use it
> to find the "good stuff" on your site and more than one person has been
> bitten by this.
>



-- 
D!
Darin W. Smith
AIM: JediGrover