> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of
> Thomas Dowling
> Sent: Wednesday, September 02, 2009 10:25 AM
> To: [log in to unmask]
> Subject: Re: [CODE4LIB] FW: PURL Server Update 2
> 
> The III crawler has been a pain for years and Innovative has shown no
> interest
> in cleaning it up.  It not only ignores robots.txt, but it hits target
> servers
> just as fast and hard as it can.  If you have a lot of links that a lot
> of III
> catalogs check, its behavior is indistinguishable from a DOS attack.  (I
> know
> because our journals server often used to crash about 2:00am on the
> first of
> the month...)

I see that I didn't fully make the connection to the point I was 
making... which is that there are hardware solutions to these 
issues rather than using robots.txt or sitemap.xml.  If a user 
agent is a problem, then network folks should change the router 
to ignore the user agent or reduce the number of requests it is
allowed to make to the server.

In the case you point to with III hitting the server as fast as 
it can and it looking like a DOS attack to the network which
caused the server to crash, then 1) the router hasn't been setup 
to impose throttling limits on user agents, and 2) the server 
probably isn't part of a server farm that is being load balanced.

In the case of GPO, they mentioned or implied, that they were
having contention issues with user agents hitting the server
while trying to restore the data.  This contention could be
mitigated by imposing lower throttling limits in the router on 
user agents until the data is restored and then raising the 
limits back to the whatever their prescribed SLA (service level
agreement) was.

You really don't need to have a document on the server to tell 
user agents what to do.  You can and should impose a network 
policy on user agents which is far better solution in my opinion.


Andy.