Hi Mark,

Actually, the sitemap.org protocol allows for a sitemap to include
references to multiple child sitemaps
http://www.sitemaps.org/protocol.html#index.

Which is what we did at my former employer:
http://digitalcollections.library.gsu.edu/sitemap/sitemap.xml

And thus the robots.txt only includes a single sitemap:
http://digitalcollections.library.gsu.edu/robots.txt
When we add extra collections, it just goes into the sitemap.xml, so we are
not continuously updating the robots.txt.

Chad



On Fri, Feb 1, 2013 at 11:33 AM, Sullivan, Mark V <[log in to unmask]>wrote:

> Jason,
>
> You may want to allow people just to give you the robots.txt file which
> references the sitemap.  I also register the sitemaps individually with the
> big search engines for our site, but I found that very large sitemaps
> aren't processed very well.  So, for our site I think I limited the number
> of items per sitemap to 40,000.  Which results in ten sitemaps for the
> digital objects and an additional sitemap for all the collections.
>
> http://ufdc.ufl.edu/robots.txt
>
> Or else perhaps give more boxes, so we can include all the sitemaps
> utilized in our systems.
>
> Cheers!
>
> Mark
>
>
> Mark V Sullivan
> Digital Development and Web Coordinator
> Technology and Support Services
> University of Florida Libraries
> 352-273-2907 (office)
> 352-682-9692 (mobile)
> [log in to unmask]
>
>
>
> ________________________________________
> From: Code for Libraries [[log in to unmask]] on behalf of Jason
> Ronallo [[log in to unmask]]
> Sent: Friday, February 01, 2013 11:14 AM
> To: [log in to unmask]
> Subject: [CODE4LIB] digital collections sitemaps
>
> Hi,
>
> I've seen registries for digital collections that make their metadata
> available through OAI-PMH, but I have yet to see a listing of digital
> collections that just make their resources available on the Web the
> way the Web works [1]. Sitemaps are the main mechanism for listing Web
> resources for automated crawlers [2]. Knowing about all of these
> various sitemaps could have many uses for research and improving the
> discoverability of digital collections on the open Web [3].
>
> So I thought I'd put up a quick form to start collecting digital
> collections sitemaps. One required field for the sitemap itself.
> Please take a few seconds to add any digital collections sitemaps you
> know about--they don't necessarily have to be yours.
>
>
> https://docs.google.com/spreadsheet/viewform?formkey=dE1JMDRIcXJMSzJ0YVlRaWdtVnhLcmc6MQ#gid=0
>
> At this point I'll make the data available to anyone that asks for it.
>
> Thank you,
>
> Jason
>
> [1] At least I don't recall seeing such a sitemap registry site or
> service. If you know of an existing registry of digital collections
> sitemaps, please let me know about it!
> [2] http://www.sitemaps.org/ For more information on robots see
> http://wiki.code4lib.org/index.php/Robots_Are_Our_Friends
> [3] For instance you can see how I've started to investigate whether
> digital collections are being crawled by the Common Crawl:
> http://jronallo.github.com/blog/common-crawl-url-index/
>