>
> One of the questions this raises is what we are/aren't allowed to do in
> terms of harvesting full-text. While I realise we could get into legal
> stuff here, at the moment we want to put that question to one side. Instead
> we want to consider what Google, and other search engines, do, the
> mechanisms available to control this, and what we do, and the equivalent
> mechanisms - our starting point is that we don't feel we should be at a
> disadvantage to a web search engine in our harvesting and use of repository
> records.
>
> Of course, Google and other crawlers can crawl the bits of the repository
> that are on the open web, and 'good' crawlers will obey the contents of
> robots.txt
> We use OAI-PMH, and while we often see (usually general and sometimes
> contradictory) statements about what we can/can't do with the contents of a
> repository (or a specific record), it feels like there isn't a nice simple
> mechanism for a repository to say "don't harvest this bit".
>

I would argue there is -- the whole point of OAI-PMH is to make stuff
available for harvesting. If someone goes to the trouble of making things
available via a protocol that exists only to make things harvestable and
then doesn't want it harvested, you can dismiss them as being totally
mental.

OAI-PMH runs on top of HTTP, so anything robots.txt already applies -- i.e.
if they want you to crawl metadata only but not download the objects
themselves because they don't want to deal with the load or bandwidth
charges, this should be indicated for all crawlers.


kyle

-- 
----------------------------------------------------------
Kyle Banerjee
Digital Services Program Manager
Orbis Cascade Alliance
[log in to unmask] / 503.999.9787