On Sun, Oct 2, 2011 at 10:32 PM, Ken Irwin <[log in to unmask]> wrote:
> 1. respect robots.txt

Respecting robots.txt is key. It's normally easy to eyeball, but when
in doubt something like robotparser [1] can be helpful. A bit of
reflection on the Golden Rule probably is probably more important than
pondering the legality of what you are doing.

//Ed

[1] http://docs.python.org/library/robotparser.html
[2] http://en.wikipedia.org/wiki/The_Golden_Rule