>>> why local library catalog records do not show up in search results?

Basically, most OPACs are crap. :-) There are still some that that
don't provide persistent links to record pages, and most are designed
so that the user has a "session" and gets kicked out after 10 minutes
or so.

These issues were part of Tim Spalding's message that as well as
joining web 2.0, libraries also need to join web 1.0.
http://vimeo.com/user2734401

>> We don't allow crawlers because it has caused serious performance issues in the past.

Specifically (in our case at least), each request creates a new
session on the server which doesn't time out for about 10 minutes,
thus a crawler would fill up the system's RAM pretty quickly.

> You can use Crawl-delay:
> http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive
>
> You can set Google's crawl rate in Webmaster Tools as well.

I've had this suggested before and thought about it, but never had it
high up enough in my list to test it out. Has anyone actually used the
above to get a similar OPAC crawled successfully and not brought down
on its knees?

David