I've been looking at the logs for our OAI server and I'd like to appeal to
those harvesting over OAI to put URLs into the user agent string. Putting
the name of your project into the user agent string seems like a great way
to build profile. It also avoids the situation where the easiest way to
contact you is via the contacts associated with your DNS block.
For reference, these are some of the user agent strings I'm seeing
(standard browser strings removed):
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
"Net::OAI::Harvester"
"Googlebot/2.1 (+http://www.google.com/bot.html)"
"OAIHarvester/2.0"
"Jakarta Commons-HttpClient/3.1"
"Mozilla/5.0 (compatible; Baiduspider/2.0; +
http://www.baidu.com/search/spider.html)"
"Celestial/3.02"
"WorldCat Digital Collection Gateway from OCLC.org"
"Apache-HttpClient/4.0.1 (java 1.5)"
"lwp-trivial/1.41"
"OAIGet-1.12"
"DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +
http://www.google.com/bot.html)"
"Mozilla/5.0 (compatible; Sosospider/2.0; +
http://help.soso.com/webspider.htm)"
"yacybot (freeworld/global; amd64 Linux 3.2.0-36-generic; java 1.6.0_27;
"OAIHarvesterObj 31 University of Illinois Library"
"PKPHarvester/2.x"
"OAI Harvester/1.0; FS Consulting, Inc."
"Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
"Typhoeus - https://github.com/typhoeus/typhoeus"
....
cheers
stuart
|