 |
 |
Andrew Houghton wrote: > In the case of GPO, they mentioned or implied, that they were having > contention issues with user agents hitting the server while trying to > restore the data. This contention could be mitigated by imposing > lower throttling limits in the router on user agents until the data is > restored and then raising the limits back to the whatever their > prescribed SLA (service level > agreement) was. The GPO tech I spoke with mentioned this contention issue explicitly. He had just emerged from a meeting on the PURL problem (which meeting sounded like an all-hands on deck affair). He mentioned that there had been discussion of the contention issue in the meeting but that they had decided not to block the offending IPs (whether because they could not do so effectively in time or because of a philosophical issue I did not inquire). Throttling the user agents was not mentioned to me as a possiblity. In fact, unless I'm mistaken, the PURL server does appear to be completely inaccessible now, in advance of advertised downtime this afternoon 5-7 EST Jonathan LeBreton Sr. Associate University Librarian Temple University Libraries voice: 215-204-8231 fax: 215-204-5201 email: [log in to unmask] email: [log in to unmask] > -----Original Message----- > From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of > Houghton,Andrew > Sent: Wednesday, September 02, 2009 11:27 AM > To: [log in to unmask] > Subject: Re: [CODE4LIB] FW: PURL Server Update 2 > > > From: Code for Libraries [mailto:[log in to unmask]] On Behalf > Of > > Thomas Dowling > > Sent: Wednesday, September 02, 2009 10:25 AM > > To: [log in to unmask] > > Subject: Re: [CODE4LIB] FW: PURL Server Update 2 > > > > The III crawler has been a pain for years and Innovative has shown no > > interest > > in cleaning it up. It not only ignores robots.txt, but it hits > target > > servers > > just as fast and hard as it can. If you have a lot of links that a > lot > > of III > > catalogs check, its behavior is indistinguishable from a DOS attack. > (I > > know > > because our journals server often used to crash about 2:00am on the > > first of > > the month...) > > I see that I didn't fully make the connection to the point I was > making... which is that there are hardware solutions to these > issues rather than using robots.txt or sitemap.xml. If a user > agent is a problem, then network folks should change the router > to ignore the user agent or reduce the number of requests it is > allowed to make to the server. > > In the case you point to with III hitting the server as fast as > it can and it looking like a DOS attack to the network which > caused the server to crash, then 1) the router hasn't been setup > to impose throttling limits on user agents, and 2) the server > probably isn't part of a server farm that is being load balanced. > > In the case of GPO, they mentioned or implied, that they were > having contention issues with user agents hitting the server > while trying to restore the data. This contention could be > mitigated by imposing lower throttling limits in the router on > user agents until the data is restored and then raising the > limits back to the whatever their prescribed SLA (service level > agreement) was. > > You really don't need to have a document on the server to tell > user agents what to do. You can and should impose a network > policy on user agents which is far better solution in my opinion. > > > Andy.