I built a Queryset class using pysolr that does this for one of my projects. It's not available as a standalone package, but the code is here:

https://github.com/unt-libraries/catalog-api/blob/master/django/sierra/utils/solr.py

Django-Haystack does this as well, but I'm not sure how usable Haystack is outside of Django. In fact, what I built here was an attempt to re-implement parts of Haystack's SearchQuerySet interface using something a bit more barebones. I found in this particular case Haystack was overkill for my needs and actually added quite a bit of overhead to Solr searches, but its interfaces are perfect for use in Django where you want to use Solr in place of Django models (e.g., using a Solr queryset in place of Django's QuerySets). I was able to shave a few hundred milliseconds off search requests this way. And since this is in context of a web API, that was important for me.

I guess it works about like you would think. After you instantiate a utils.solr.Queryset object (passing connection details, a page_by parameter, and the kwargs you want to send to Solr), you access individual results using the Queryset object like you would a list. Behind the scenes, it sends queries to Solr as needed to fetch the result (or results) you're accessing and caches the last result set. It only sends a query when you try to access a result outside the cached result set.

Some of the filters still need some work, and it *is* barebones--features like highlighting and faceting aren't implemented at all--but it works for what I use it for, and it shows how you might go about abstracting Solr paging logic with pysolr. (FWIW I haven't tried any of the other numerous Python modules for interacting with Solr so I'm not sure if others do something similar...)

Jason


> -----Original Message-----
> From: Code for Libraries [mailto:[log in to unmask]] On Behalf Of
> Tod Olson
> Sent: Thursday, September 01, 2016 5:28 AM
> To: [log in to unmask]
> Subject: Re: [CODE4LIB] python for solr
> 
> Exactly! The question is whether there is a python solr library that
> provides a layer of abstraction over that paging logic.
> 
> -Tod
> 
> Sent from from the æther.
> 
> > On Sep 1, 2016, at 04:59, Andrew Hankinson
> <[log in to unmask]> wrote:
> >
> > Solr itself has an internal limit to the number of results you can
> return on a single page (I think it is 1000) and AFAIK always returns a
> paged result. For speed and memory usage over large result sets it would
> probably be most efficient to build in paging logic.
> >
> >> On Aug 31, 2016, at 10:45 PM, Tod Olson <[log in to unmask]> wrote:
> >>
> >> On a related note, do any of the libraries allow the user to iterate
> over a large result set without having to be aware of repeated calls,
> incrementing the start parameter, and that sort of bookkeeping?
> >>
> >> It seems like someone must have built an iterator to hide that when
> you're trying to sift through a large number of hits.
> >>
> >> -Tod
> >>
> >>> On Aug 31, 2016, at 4:09 PM, Rhoads, Joseph
> <[log in to unmask]> wrote:
> >>>
> >>> I've used several of these.  I like the interface of mysolr but (as
> >>> mentioned) it hasn't been updated in a while.
> >>>
> >>> pysolr is fairly up to date (v3.5 came out in May this year), and is
> used
> >>> in django-haystack for the solr backend.
> >>> https://github.com/django-haystack/pysolr
> >>>
> >>> Haystack itself is great if you want an ORM-like interface for solr
> and use
> >>> django.
> >>> https://github.com/django-haystack/django-haystack
> >>>
> >>> -Joseph
> >>>
> >>>
> >>>
> >>>> On Wed, Aug 31, 2016 at 3:42 PM, Chris Gray <[log in to unmask]>
> wrote:
> >>>>
> >>>> I haven't done much of that but you can submit documents via the
> API and
> >>>> have them indexed (and processed by Tika).  Once you understand how
> to do
> >>>> that, you might find that you can do everything you want to do.
> >>>>
> >>>> An alternative would be reading the source of one of those
> libraries.  In
> >>>> the list you referenced, the only mention of inserting documents
> was for
> >>>> sunburnt.  I would be inclined to look there first, especially
> since it
> >>>> mentions a pythonic interface to Solr.
> >>>>
> >>>> A good, and amusing, cautionary tale about overwritten Python
> libraries is
> >>>> at https://www.youtube.com/watch?v=o9pEzgHorH0.
> >>>>
> >>>> Chris
> >>>>
> >>>>
> >>>>> On 2016-08-31 03:28 PM, Eric Lease Morgan wrote:
> >>>>>
> >>>>> On Aug 31, 2016, at 3:25 PM, Chris Gray <[log in to unmask]>
> wrote:
> >>>>>
> >>>>> Okay, there are SO many Python libraries [1] for Solr, and I'd
> like to
> >>>>>>> know which one is the most popular (not necessarily the "best").
> >>>>>> What do you want to do with it?
> >>>>>>
> >>>>>> I didn't feel the need to even look for a Python library for my
> needs.
> >>>>>> I use Python to submit searches to the Solr web API and consume
> the results
> >>>>>> as JSON.
> >>>>>
> >>>>> Good question. I want to add documents to a Solr index, and I want
> to
> >>>>> query the same index. Hmmm. -Eric M.
> >>