The URBE Consortium is planning to apply fail2ban rules to detect and block high rate accesses.
sb

-- 
Dott. Stefano Bargioni
Pontificia Universita' della Santa Croce - Roma
Vicedirettore della Biblioteca
<mailto:[log in to unmask]>        <http://www.pusc.it>
--- "Non refert quam multos habeas libros, sed bonos" (Seneca) ---


> On 9 Apr 2024, at 00:15, Bruce Orcutt <[log in to unmask]> wrote:
> 
> also following as also been find some crawlers  being less and less behaved, ignoring robots.txt, scanning fast enough to impact performance, etc.  used to just be a handful of badly behaving bots but definitely growing of late.
> 
> Bruce Orcutt
> UTSA Libraries: Systems
> (210) 458- 6192
> ________________________________
> From: Code for Libraries <[log in to unmask]> on behalf of Jason Casden <[log in to unmask]>
> Sent: Monday, April 8, 2024 3:30:14 PM
> To: [log in to unmask] <[log in to unmask]>
> Subject: [EXTERNAL] Re: [CODE4LIB] blocking GPTBot?
> 
>  **EXTERNAL EMAIL**
>  This email originated outside of The University of Texas at San Antonio.
>  Please exercise caution when clicking on links or opening attachments.
> 
> 
> 
> Thanks for bringing this up, Eben. We've been having a horrible time with
> these bots, including those from previously fairly well-behaved sources
> like Google. They've caused issues ranging from slow response times and
> high system load all the way up to outages for some older systems. So far,
> our systems folks have been playing whack-a-mole with a combination of IP
> range blocks and increasingly detailed robots.txt statements. A group is
> being convened to investigate more comprehensive options so I will be
> watching this thread closely.
> 
> Jason
> 
> On Mon, Apr 8, 2024 at 4:18 PM Eben English <[log in to unmask]> wrote:
> 
>> Hi all,
>> 
>> I'm wondering if other folks are seeing AI and/or ML-related crawlers like
>> GPTBot accessing your library's website, catalog, digital collections, or
>> other sites.
>> 
>> If so, are you blocking or disallowing these crawlers? Has anyone come up
>> with any policies around this?
>> 
>> We're debating whether to allow these types of bots to crawl our digital
>> collections, many of which contain large amounts of copyrighted or "no
>> derivatives"-licensed materials. On one hand, these materials are available
>> for public view, but on the other hand the type of use that GPTBot and the
>> like are after (integrating the content into their models) could be
>> characterized as creating a derivative work, which is expressly
>> discouraged.
>> 
>> Thanks,
>> 
>> Eben English (he/him/his)
>> Digital Repository Services Manager
>> Boston Public Library
>>