Grr..

:1,$s/work/word/g

(I blame FRBR ;))

e


On Fri, Feb 24, 2012 at 4:52 PM, Ian Ibbotson <[log in to unmask]>wrote:

> Sorry.. late to the discussion...
>
> Isn't this a little apples and oranges?
>
> Surely robots.txt exists because many static resources are served directly
> from a tree structured filesystem?
>
> (Nearly) all OAI requests are responded to by specific service
> applications which are perfectly capable of deciding, on a resource by
> resource basis if an anonymous user should or should not see a given
> resource. As has been said, why would you list a resource in OAI if you
> didn't think **someone** would find it useful. If you want to take
> something out of circulation, you mark it deleted so that clients
> connecting for updates know it should be removed.
>
> OAI isn't about fully enumerating a tree on every visit to see whats new,
> it's about a short and efficient visit to say "What, if anything, has
> changed since I was last here". I don't want to have to walk an entire
> repository of 3 million items to discover item 2999999 was deleted.. I want
> a message to say "Oh, item 2999999 was removed on X".
>
> Not sure my thinking is entirely clear on this, but I don't see the 2
> things as being that similar. (Apart from the work harvest being used in
> the context).
>
> P.S. We never got that beer Owen!
>
> e
>
>
> Ian Ibbotson
> Director
> Knowledge Integration Ltd
> 35 Paradise Street, Sheffield. S3 8PZ
> T: 0114 273 8271
> M: 07968 794 630
> W: http://www.k-int.com
>
>
>
> On Fri, Feb 24, 2012 at 4:31 PM, Owen Stephens <[log in to unmask]> wrote:
>
>> Thanks both...
>>
>> Kyle said: "If someone goes to the trouble of making things available via
>> a protocol that exists only to make things harvestable and
>> then doesn't want it harvested, you can dismiss them ..."
>>
>> True - but that's essentially what Southampton's configuration seems to
>> say.
>>
>> Thomas said: "The M in PMH still stands for Metadata, right?  So opening
>> an OAI-PMH server implicitly says you're willing to share metadata.  I can
>> certainly sympathize with sites wanting to do that but not necessarily
>> wanting to offer anything more than "normal" end-user access to full text."
>>
>> This is a fair point - but I've yet to see an example of a robots.txt
>> file that makes this distinction - that is, in general Google is not being
>> told to not crawl and cache pdfs, while being granted explicit permission
>> to crawl the metadata, no matter what the OAI-PMH situation.
>>
>> Kyle said: "OAI-PMH runs on top of HTTP, so anything robots.txt already
>> applies -- i.e. if they want you to crawl metadata only but not download
>> the objects themselves because they don't want to deal with the load or
>> bandwidth charges, this should be indicated for all crawlers."
>>
>> OK - this suggests a way forward for me. Although I don't think we can
>> regard robots.txt applying across the board for OAI-PMH (as in the
>> Southampton example, the OAI-PMH endpoint is disallowed by robots.txt), it
>> seems to make sense that given a resource identifier in the metadata we
>> could use robots.txt (and I guess potentially x-robots-tag, assuming most
>> of the resources are not simple html) to see whether a web crawler is
>> permitted to crawl it, and so make the right decision about what we do.
>>
>> That sounds vaguely sensible (although I'm still left thinking, maybe we
>> should just use a web crawler and ignore OAI-PMH but I guess this was we
>> maybe get the best of both worlds).
>>
>> Thanks again (and of course further thoughts welcome)
>>
>> Owen
>>
>> Owen Stephens
>> Owen Stephens Consulting
>> Web: http://www.ostephens.com
>> Email: [log in to unmask]
>> Telephone: 0121 288 6936
>>
>> On 24 Feb 2012, at 14:45, Thomas Dowling wrote:
>>
>> > On 02/24/2012 09:25 AM, Kyle Banerjee wrote:
>> >
>> >>> We use OAI-PMH, and while we often see (usually general and sometimes
>> >>> contradictory) statements about what we can/can't do with the
>> contents of a
>> >>> repository (or a specific record), it feels like there isn't a nice
>> simple
>> >>> mechanism for a repository to say "don't harvest this bit".
>> >>>
>> >>
>> >> I would argue there is -- the whole point of OAI-PMH is to make stuff
>> >> available for harvesting. If someone goes to the trouble of making
>> things
>> >> available via a protocol that exists only to make things harvestable
>> and
>> >> then doesn't want it harvested, you can dismiss them as being totally
>> >> mental.
>> >
>> > The M in PMH still stands for Metadata, right?  So opening an OAI-PMH
>> > server implicitly says you're willing to share metadata.  I can
>> certainly
>> > sympathize with sites wanting to do that but not necessarily wanting to
>> > offer anything more than "normal" end-user access to full text.
>> >
>> > That said, in a world with unfriendly bots, the repository should still
>> be
>> > making informed choices about controlling full text crawlers
>> (robots.txt,
>> > meta tags, HTTP cache directives, etc etc.).
>> >
>> >
>> > --
>> > Thomas Dowling
>> > [log in to unmask]
>>
>
>