Hi Shannon,

At our institution we have employed a great diversity of techniques, including experimentation with Anubus and Cloudflare Turnstile — depending on the nature of the service. But, in general, the approach has been a combination of a) monitoring traffic more effectively and blocking where necessary, b) rate limiting, and c) web application firewalls (WAFs), as above.

On WAFs: We have deployed a WAF on only two services, and only out of necessity. And I would say that they have indeed been effective, though they are not a silver bullet and the other techniques mentioned remain necessary. I dislike using WAFs because it is difficult to anticipate honourable bot actors in our domain, and it is highly likely that indiscriminate blocking of 'welcome' bots is frequently occurring, despite the ingenuity of the WAF approach. My understanding is that Cloudflare Turnstile is free to universities (at least in the UK); my institution has a bunch of other Cloudflare products, so I imagine we pay for Turnstile indirectly. Anubus is OS, as you note. So is 'Go Away', though I don't have experience of using it — it might be superior to Anubus.

Apologies if you have already checked it out, but COAR have a website dedicated to 'dealing with the bots'. It includes some useful advice, suggested strategies, and suggested solutions, vendors, etc. See: https://dealing-with-bots.coar-repositories.org/

Hope some of this helps!

Cheers

George

--

Dr George Macgregor | Assistant Director – Digital Library

Information Services | University of Glasgow

Web: https://purl.org/g3om4c  | Fediverse: @[log in to unmask]<https://code4lib.social/@g3om4c>
[ORCID logo]orcid.org/0000-0002-8482-3973<http://orcid.org/0000-0002-8482-3973>

Mobile: +44 (0)7977 858281
--

The University of Glasgow is a registered Scottish charity: Registration Number SC004401

--

________________________________
From: Code for Libraries <[log in to unmask]> on behalf of Lucky, Shannon <[log in to unmask]>
Sent: Thursday, April 30, 2026 19:35
To: [log in to unmask] <[log in to unmask]>
Subject: [CODE4LIB] Dealing with bot traffic - what tools/services are you using?

[Some people who received this message don't often get email from [log in to unmask] Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Hi all,

I am curious what methods folks are using to deal with aggressive AI harvesting on websites - particularly digital project sites. Many of our servers are being hammered with traffic that impacts our service delivery and the methods we have been using cannot keep up.

Specifically I am wondering who is using services like Cloudflare or implementing OS solutions like Anubis, or are you using something else? I'm gathering information about what services or methods are being using at academic libraries hosting DH/digital projects so we can look at investing in some kind of service or process solution.

What are you using? Are you happy with it? What kinds of costs are associated?)


Shannon Lucky, MLIS MA

she/her

Associate Librarian


University of Saskatchewan

University Library

Ph: 306-966-2740

ORCID 0000-0001-9134-8560<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Forcid.org%2F0000-0001-9134-8560&data=05%7C02%7CGeorge.Macgregor%40glasgow.ac.uk%7Cc38c746aa30b4c5e961208dea6e75509%7C6e725c29763a4f5081f22e254f0133c8%7C1%7C0%7C639131709662824787%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=G53Egsj1Plb2pkWMrgtvYoGIlIuCwQWq%2BpuNJ%2FG0MoE%3D&reserved=0<https://orcid.org/0000-0001-9134-8560>>


I acknowledge that I live and work on Treaty 6 Territory and the Homeland of the Métis. We pay our respect to the First Nations and Métis ancestors of this place and reaffirm our relationship with one another.