We're not an academic library, but we have a couple of what we call "Varnish servers" out in front of the cluster of machines that runs ID.LOC.GOV. Varnish [1] caches requests, of course, which takes a little pressure off the backend machines, but it also gives us decent control over managing traffic. (We also have cloudflare out front, but Varnish gives us who manage the service finer control.) We have all types of rules to block known bots, especially AI bots, as well as individual IPs and entire ranges of them. It's becoming harder and harder to manage because I can't help but think there are a lot of bots masquerading as human users, which is to say that they are using common browser User Agents versus identifying themselves. In any event, it's pretty much a game of whac-a-mole at this point.
Anyways, you are not alone. We too have been absolutely hammered now for pretty much a year plus; simply a silly amount of traffic that aggravatingly diminishes the service for 'real' users.
HTH,
Kevin
[1] https://www.varnish-software.com/products/varnish-cache/
--
Kevin Ford
Library of Congress
Washington, DC
-----Original Message-----
From: Code for Libraries <[log in to unmask]> On Behalf Of Lucky, Shannon
Sent: Thursday, April 30, 2026 14:36
To: [log in to unmask]
Subject: [CODE4LIB] Dealing with bot traffic - what tools/services are you using?
CAUTION: This email message has been received from an external source. Please use caution when opening attachments, or clicking on links.
Hi all,
I am curious what methods folks are using to deal with aggressive AI harvesting on websites - particularly digital project sites. Many of our servers are being hammered with traffic that impacts our service delivery and the methods we have been using cannot keep up.
Specifically I am wondering who is using services like Cloudflare or implementing OS solutions like Anubis, or are you using something else? I'm gathering information about what services or methods are being using at academic libraries hosting DH/digital projects so we can look at investing in some kind of service or process solution.
What are you using? Are you happy with it? What kinds of costs are associated?)
Shannon Lucky, MLIS MA
she/her
Associate Librarian
University of Saskatchewan
University Library
Ph: 306-966-2740
ORCID 0000-0001-9134-8560<https://urldefense.us/v3/__https://orcid.org/0000-0001-9134-8560__;!!MrYkk0_46kUzGAu-DfDRZGQ!bjkp5G1qG8FLYuGRKenKf4dwaI6-uhziERQ3nwAszCSW3h-_vrWkxhcFr0q7X_xLU-Kh_6YNj2dJxAejJixt_kX6Bv8$ >
I acknowledge that I live and work on Treaty 6 Territory and the Homeland of the Métis. We pay our respect to the First Nations and Métis ancestors of this place and reaffirm our relationship with one another.
|
