 |
 |
On Fri, Feb 24, 2012 at 6:50 AM, Owen Stephens <[log in to unmask]> wrote: > One obvious alternative that I keep coming back to is 'forget OAI-PMH, just crawl the web' ... Owen, I'd like to bring this back to your suggestion to just forget OAI-PMH and crawl the web. I think that's probably the long-term way forward. I've written some about how new aggregators might crawl the web for content. [1] The content that is then available to you would be greater than just those sites which happen to have an OAI-PMH gateway still. As long as the content is up on the Web in some form you could then make some use of it. There are some problems, though with crawling the web. For instance, how do you determine which pages are the item-level resources you're interested in? I think there are some simple solutions to that problem like URL templates and sitemaps. [2] These are things that will help make the sites from which you harvest more discoverable through the search engines as well. Google no longer uses OAI-PMH. You'd then be asking folks to optimize for the Web which would bring them added benefits, rather than optimizing just for aggregators that happen to know they have an OAI-PMH gateway. Another concern is curating what sites get crawled, and I think this is where having a site with an API which holds information about collections makes things easier. [3] Despite the challenges, crawling sites is the Web way of doing things. Then you can use robots.txt and meta tags to determine what you can't crawl or index. There's also a lot happening these days around open crawl data like the Common Crawl [3] and Web Data Commons [4]. I've written a bit about how this new, open, big Web data could be used by aggregators. [6] You may not have to crawl the Web yourself, but could reuse crawl data. In the short term as a transition, you could probably begin by merging the metadata-based approach of harvesting over OAI-PMH with Web crawl data. Until more sites are embedding semantic markup on their pages, having the fielded Dublin Core metadata can help build better applications than you might with just using natural language processing of the crawl data. But if all you have for a site is your crawl data, then at least you can do something with it and not exclude it. If you do decide to go in the direction of crawling the Web, I'd be interested in talking with you more about it. Jason [1] http://jronallo.github.com/blog/dpla-strawman-technical-proposal/ [2] http://jronallo.github.com/blog/solving-the-item-level-problem-on-the-web/ [3] Collection Achievements and Profiles System background: go.ncsu.edu/dplacaps and prototype: http://capsys.herokuapp.com/ [4] http://commoncrawl.org/ [5] http://page.mi.fu-berlin.de/muehleis/ccrdf/ [6] http://jronallo.github.com/blog/code4lib-2012-lightning-talk-that-wasnt/