On Jun 18, 2026, at 10:34 AM, Charlow, Aurora <[log in to unmask]> wrote:

> I'd look at this repo as a first stop: https://github.com/iipc/awesome-web-archiving#tools--software


On Jun 18, 2026, at 11:22 AM, Ed Summers <[log in to unmask]> wrote:

> Down at the bottom of the Awesome page that Aurora mentioned is a Public Data section, which includes probably the largest publicly available pool of WARC data: CommonCrawl:
> 
>    https://data.commoncrawl.org/
> 
> The other one is the End of Term Archive:
> 
>    https://eotarchive.org/
> 
> Both projects hast their WARC data on Amazon S3.
> 
> The Internet Archive is probably sitting on the largest pile of WARC data, but they generally don’t make it available to the public.
> 
> You’ll know you’ve caught the web archiving bug when you start creating your own WARC data using some of the tools on that Awesome list. Browsertrix [1] is a fun one to try: open source and available as a service. ArchiveBox [2] is another open source project for archiving the web that will write WARC files. Probably one of the most CLI friendly way to create WARC data is using wget to crawl some region of the web. Or you can try Harvard’s scoop tool. There are a lot of fun things to try!
> 
> //Ed
> 
> [1] https://docs.browsertrix.com/
> [2] https://archivebox.io/
> [3] https://www.gnu.org/software/wget/manual/
> [4] https://github.com/harvard-lil/scoop#readme



Thank you for the prompt replies, but I still don't see any URLs pointing to warc files, sans warc files terabytes or petabyte in size. There are many descriptions of sites that have been converted into warc files, but no access to the warc files themselves. I'd like to play with a dozen or so warc files, not of my own design, that are in the few gigabyte size range. What am I missing? --E "Still Embarrassed" M