 |
 |
On Sun, Oct 2, 2011 at 10:32 PM, Ken Irwin <[log in to unmask]> wrote: > 1. respect robots.txt Respecting robots.txt is key. It's normally easy to eyeball, but when in doubt something like robotparser [1] can be helpful. A bit of reflection on the Golden Rule probably is probably more important than pondering the legality of what you are doing. //Ed [1] http://docs.python.org/library/robotparser.html [2] http://en.wikipedia.org/wiki/The_Golden_Rule