On Thu, Feb 23, 2012 at 2:07 PM, Baksik, Corinna M.
<[log in to unmask]> wrote:
> We don't allow crawlers because it has caused serious performance issues in the past.
There are simple solutions that may help for the performance problem.
You can use Crawl-delay:
You can set Google's crawl rate in Webmaster Tools as well.