On Oct 3, 2011 9:19 AM, "Ed Summers" <[log in to unmask]> wrote:

> On Sun, Oct 2, 2011 at 10:32 PM, Ken Irwin <[log in to unmask]> wrote:
> > 1. respect robots.txt

Disclaimer: I am not a lawyer.

Remember that robots.txt applies only to recursive web crawlers, and not to
screen-scraping per se. In cases where it does apply, it has limited legal
effect, but ignoring it is not cricket.

Important considerations are: is access to the site governed by a license
that prohibits the activity; is the content being scraped subject to
copyright, and if so, is the screen scraping covered by one of the
exceptions to exclusive rights of the copyright holder; is the
screen-scraping activity disruptive and damaging to the site being used
(trespass to chattels, etc.)?

>A bit of reflection on the Golden Rule probably is probably more important
than pondering the legality of what you are doing.

Ed invoking philosophy? With citation? (wikipedia still counts) :-p

The usual objection to the golden rule apply here- just because one has no
objection to having a screen scraper used on your own site doesn't
automatically imply that others might not wish to have their sites scraped.

Simon