On Sun, Oct 2, 2011 at 10:32 PM, Ken Irwin <[log in to unmask]> wrote:
> 1. respect robots.txt
Respecting robots.txt is key. It's normally easy to eyeball, but when
in doubt something like robotparser  can be helpful. A bit of
reflection on the Golden Rule probably is probably more important than
pondering the legality of what you are doing.