 |
 |
> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of > David Fiander > Sent: Wednesday, September 02, 2009 9:32 AM > To: [log in to unmask] > Subject: Re: [CODE4LIB] FW: PURL Server Update 2 > > If Millenium is acting like a robot in its > monthly maintenance processes, then it should be checking robots.txt. User agents are *not required* to check robots.txt nor are servers *required* to provide a robots.txt. There are no expectations for robots.txt other than a gentlemen's agreement that if a server provides one it should be consulted when any content is access from the server. However, if you have publicly accessible URIs it is highly unlikely that you would restrict access to those of URIs in your robots.txt. It kink-of defeats the purpose of the URIs being *public*. You might put those URIs in robots.txt when the URIs have been deprecated and are being redirected to another URI, e.g., you redesigned your Web site, but 1) I would argue that it would be better for your user agents to see the redirect so they can update themselves, and 2) GPO is running a PURL server, where the URIs are suppose to be *permanent* and *publicly* accessible. Robots.txt is a nice idea, but if you are having an issue with a user agent, the network folks will most likely update the router rules to block the traffic rather than let it get thru to the server. <http://www.robotstxt.org/orig.html> Andy.