On Wed, Feb 20, 2013 at 2:33 PM, Nathan Tallman <[log in to unmask]> wrote:
> @Péter: The VuFind solution I mentioned is very similar to what you use
> here. It uses Aperture (although soon to use Tika instead) to grab the
> full-text and shoves everything inside a solr index. The import is managed
> through a PHP script the crawls every URL on the sitemap. The only part I
> don't have is removing deleted, adding new, and updating changed
> webpages/files. I'm not sure how to rework the script to use a list of new
> files rather than the sitemap, but everything is on the same server so that
> should work.

Nathan,

A first step could be to record a timestamp of when a particular URL
is fetched. Then modify your PHP script to send an "If-Modified-Since"
header with the request. Assuming the target server adheres to basic
HTTP behavior, you'll get a 304 response and therefore know you don't
have to re-index that particular item.

(As an aside, could Google be ignoring items in your sitemap that it
thinks haven't changed?)

Maybe I'm misunderstanding though. The sitemap you mention has links
to html pages which then link to the PDFs? So you have to parse the
HTML to get the PDF URL? In that case, it still seems like recording
the last-fetched timestamps for the PDF URLs would be an option. I
know next to nothing about VuFind, so maybe the fetching mechanism
isn't exposed in a way to make this possible. I'm surprised it's not
already baked in, frankly.

One other thing that's confusing is the notion of "over 1000 PDFs"
taking a "long, long time". Even on fairly milquetoast hardware, I'd
expect solr to be capable of extracting and indexing 1000 PDF
documents in 20-30 minutes.

--jay