I'd be very surprised if Google _automatically_ took any notice of
anything in an HTTP header to relax protection against what they
consider harvesting of data because all HTTP headers can be set to
anything: that is, if I wanted to suck Google dry of bib data, I
could simply pretend to be forwarding requests for "real" clients
behind a NAT barrier.
But they may well investigate such cases and configure their traffic
monitoring software for known legitimate proxies.
On Wed, Mar 19, 2008 at 3:29 AM, Joe Hourcle
<[log in to unmask]> wrote:
> On Tue, 18 Mar 2008, Jonathan Rochkind wrote:
> > Wait, now ALL of your clients calls are coming from one single IP?
> > Surely that will trigger Googles detectors, if the NAT did. Keep us
> > updated though.
> I don't know what Peter's exact implementation is, but they might relax
> the limits when they see an 'X-Forwarded-For' header, or something else to
> suggest it's coming through a proxy. It used to be pretty common when
> writing rate limiting code to use X-Forwarded-For in place of HTTP_ADDR so
> you didn't accidentally ban groups behind proxies. (of course, I don't
> know if the X-Forwarded-For value is something that's not routable (in
> 10/8), or the NAT IP, so it might still look like 1 IP address behind a
> Also, by using a caching proxy (if the responses are cachable), the total
> number of requests going to Google might be reduced.
> I would assume they'd need to have some consideration for proxies, as I
> remember the days when AOL's proxy servers channeled all requests through
> less than a dozen unique IP addresses. (or at least, those were the only
> ones hitting my servers)