I've been making my headers very much like a proxy would, even sending
X-forwarded-for headers with the original client ip and such. Sending
very close to the same thing that would be sent if the user really was
using a web proxy. Seems to be keeping google happy... so far.
Jonathan
Michael Beccaria wrote:
> Not everyone will care, but I will put it in here for posterity sake and
> probably for my own reference when I forget in the future.
>
> I was having trouble getting the new google books dynamic link api to
> work right with python
> (http://code.google.com/apis/books/docs/dynamic-links.html). I was using
> the basic urllib python library with a non-working code base that looks
> like this:
>
> import urllib,urllib2
> gparams = urllib.urlencode({'bibkeys': 'ISBN:061837943',
> 'jscmd':'viewapi','callback':'mycallback'})
> g=urllib2.urlopen(url="http://books.google.com/books?%s" % gparams)
> print g.read()
>
> I was getting an http 401 error, Unauthorized. Code4lib IRC folks told
> me it was probably the headers urllib was sending, and they were right.
> I wrote code to modify the headers to make google believe I was
> requesting from firefox. The working code is below. I know most of you
> can write this stuff in your sleep, but I thought this might save
> someone like me some time in the end.
> Hope it helps,
> Mike Beccaria
> Systems Librarian
> Head of Digital Initiatives
> Paul Smith's College
> 518.327.6376
> [log in to unmask]
>
>
> import urllib,urllib2
> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor())
> params = urllib.urlencode({'bibkeys': 'ISBN:061837943',
> 'jscmd':'viewapi','callback':'mycallback'})
>
> request =
> urllib2.Request('http://books.google.com/books?bibkeys=0618379436&jscmd=
> viewapi&callback=mycallback')
> opener.addheaders = [('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT
> 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3')]
> data = opener.open(request).read()
> print data
>
>
--
Jonathan Rochkind
Digital Services Software Engineer
The Sheridan Libraries
Johns Hopkins University
410.516.8886
rochkind (at) jhu.edu
