On Thu, Feb 23, 2012 at 2:07 PM, Baksik, Corinna M.
<[log in to unmask]> wrote:
> We don't allow crawlers because it has caused serious performance issues in the past.

There are simple solutions that may help for the performance problem.

You can use Crawl-delay:
http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive

You can set Google's crawl rate in Webmaster Tools as well.

Jason