 |
 |
We've also been seeing some traffic from inconsiderate AI bots. One of my colleagues came across this site, which tracks and documents AI bots: https://darkvisitors.com/ -- Scott -- Scott Prater Digital Library Architect UW Digital Collections Center University of Wisconsin - Madison ________________________________________ From: Code for Libraries <[log in to unmask]> on behalf of Lolis, John <[log in to unmask]> Sent: Wednesday, April 10, 2024 12:15 PM To: [log in to unmask] Subject: Re: [CODE4LIB] blocking GPTBot? This *sounds* as if it should help: https://urldefense.com/v3/__https://searchengineland.com/google-extended-crawler-432636__;!!Mak6IKo!Pm6vbeyDLkzwxaEhcIBmaI0pK1d7U0GtguiIAgWmNzfNyOyR1m3n9iypyhqwZH3QxxIfNMIETf94S2_ioTtPPtfncyM$ John Lolis Coordinator of Computer Systems 100 Martine Avenue White Plains, NY 10601 tel: 1.914.422.1497 fax: 1.914.422.1452 https://urldefense.com/v3/__https://whiteplainslibrary.org/__;!!Mak6IKo!Pm6vbeyDLkzwxaEhcIBmaI0pK1d7U0GtguiIAgWmNzfNyOyR1m3n9iypyhqwZH3QxxIfNMIETf94S2_ioTtPwb7-RSk$ *“I would rather have questions that can’t be answered than answers that can’t be questioned.”* — Richard Feynman <https://urldefense.com/v3/__https://click.fourhourmail.com/5qure95xkf7hvvo93wh2/7qh7h8h05vr4zrtz/aHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvUmljaGFyZF9GZXlubWFu__;!!Mak6IKo!Pm6vbeyDLkzwxaEhcIBmaI0pK1d7U0GtguiIAgWmNzfNyOyR1m3n9iypyhqwZH3QxxIfNMIETf94S2_ioTtP3X91XJ0$ >, theoretical physicist and recipient of the Nobel Prize in Physics in 1965 On Mon, 8 Apr 2024 at 16:31, Jason Casden <[log in to unmask]> wrote: > Thanks for bringing this up, Eben. We've been having a horrible time with > these bots, including those from previously fairly well-behaved sources > like Google. They've caused issues ranging from slow response times and > high system load all the way up to outages for some older systems. So far, > our systems folks have been playing whack-a-mole with a combination of IP > range blocks and increasingly detailed robots.txt statements. A group is > being convened to investigate more comprehensive options so I will be > watching this thread closely. > > Jason > > On Mon, Apr 8, 2024 at 4:18 PM Eben English <[log in to unmask]> > wrote: > > > Hi all, > > > > I'm wondering if other folks are seeing AI and/or ML-related crawlers > like > > GPTBot accessing your library's website, catalog, digital collections, or > > other sites. > > > > If so, are you blocking or disallowing these crawlers? Has anyone come up > > with any policies around this? > > > > We're debating whether to allow these types of bots to crawl our digital > > collections, many of which contain large amounts of copyrighted or "no > > derivatives"-licensed materials. On one hand, these materials are > available > > for public view, but on the other hand the type of use that GPTBot and > the > > like are after (integrating the content into their models) could be > > characterized as creating a derivative work, which is expressly > > discouraged. > > > > Thanks, > > > > Eben English (he/him/his) > > Digital Repository Services Manager > > Boston Public Library > > >