wtf? almost every forum and website is on cloudflare now?

[Moonchild]

I can't believe all the ISP don't see these "bots" "AI scrapers"

Oh make no mistake. They would absolutely see them -- they just aren't looking.
It's primarily because most ISPs (practically all of them) have this aversion to egress filtering. What I mean with that is that, in general on the larger scale, ISPs don't want to spend some of that money they take from their subscribers to invest in filtering bad traffic originating in their network, because it impacts others outside of their organization (and not them). Instead, they focus on ingress, i.e. filtering bad traffic targeting them.
Cloudflare's turnstile is just more of that: trying to mitigate attacks on the receiving end, not the originating end. If egress filtering became enforced globally, that would solve the majority of these problems (as well as SPAM) at the root. It wouldn't even be that hard to do if there's willingness, because there can be severe consequences for ISPs if they don't play nice, like losing their peering agreements. There's just one problem: the industry doesn't want to, because their bad traffic clients are paying them good money for big data pipelines that they would lose by punishing the bad behaviour.

[browserman]

There's just one problem: the industry doesn't want to, because their bad traffic clients are paying them good money for big data pipelines that they would lose by punishing the bad behaviour.

Very well said!

[Kris_88]

I don't understand why people always talk about attacks when it comes to bots. In reality, any bot is an extension of a person. We shouldn't forget that. The purpose of using bots is important here. If a person wants to get a copy of a website, for example, to avoid the risk of it suddenly disappearing (everyone's familiar with this situation), and it contains valuable information, then this desire is not malicious. Moreover, I've seen links on some websites for downloading all the contents as an archive, and I think that's good form. It would be good to create a special line in robots.txt for this purpose. If the goal is to deliberately overload the server with requests, then that's malicious intent, and that's a completely different matter. If search engines crawl a website to index content, or a robot collects information to train AI, there's nothing malicious about it. After all, most people use search engines, and many use AI. I don't see anything wrong with that.

[Moonchild]

The purpose of using bots is important here. If a person wants to get a copy of a website, for example, to avoid the risk of it suddenly disappearing (everyone's familiar with this situation), and it contains valuable information, then this desire is not malicious.

The desire isn't malicious, but the execution of fulfilling that desire is.
The right thing to do in that case is not to indiscriminately launch scraper bots at the site, but rather to contact the site owner and ask for a copy.
You'll be surprised how simple things can get if you just politely ask for a copy of public information for safekeeping.

[Kris_88]

You'll be surprised how simple things can get if you just politely ask for a copy of public information for safekeeping.

Contacting the owner and asking for help is a fairly complex task to automate. It involves at least one person (the website owner), and their response is human-readable, meaning it's not formalized for automated processing. Multiply this by the number of websites, and the task becomes so labor-intensive that it's practically insurmountable. I think people will eventually come to some kind of formalized, global solution. And by the way, I'm curious: when the offline copy of old extensions from the Mozilla website was created, was it done by contacting the website owners and Mozilla preparing an archive? Somehow, I don't think that was the case.

[BenFenner]

If search engines crawl a website to index content, or a robot collects information to train AI, there's nothing malicious about it.

Image the site owner has made it clear to the perpetrator(s) that this usage is undesirable and against their wishes. Like I have for my site. Yet the malicious attempts continue.

Image that. Different people find different things malicious.

[Kris_88]

Image the site owner has made it clear to the perpetrator(s) that this usage is undesirable and against their wishes.

Why? A search engine will download your site once, but then it will show it to many people. Is that bad? Yes, I understand that maybe you created your site for other purposes, for example, for a small circle of friends, and you're not interested in expanding your audience. But overall, this is rather unusual. Search engines are beneficial for websites. I can also imagine that you don't use search engines yourself. That is, you don't use them at all, and it wouldn't bother you at all if every website on the internet decided to ban their indexing. But that's also unusual...

And, as I've already said, every bot is an embodiment of its creator. If anyone can access your site, it essentially makes no difference whether they do so by clicking links with their mouse or by setting up some automated process to do it for them.

[Mæstro]

There are real differences. If someone visits a site himself, he actually sees whatever the webmaster has written and wants to share with the world. A bot crawling the site cannot do this. It is a soulless, automated request which copies the page into a database which may or may not ever actually be read. For small servers especially, with limited bandwidth*, a webmaster might wish to reserve his traffic for real people.

Again, different people have different purposes. Indexing and archival are generally regarded as benign here, although there might be reasons somebody wants his site unlisted, and a general claim like ‘search engines are beneficial for websites’ does not trump these specific purposes. Privacy might be one reason (cf the right to be forgotten), but one is not required to justify one’s wish to be excluded to strangers, be they search engine proprietors or others. But I think the strongest objection is that against scraping one’s site ‘to train AI’. Many people, I among them, are categorically opposed to this technology’s existence and abstain completely from it.† We regard any use of our words or webpages for LLM or MMM as abuse, an offence upon our moral rights, and bots intending that malicious. It does not matter one jot if many people use it.

*Bandwidth shortages still happen today. Just last week, an independent TRPG site which has existed for over twenty years was down because the webmaster had run out of bandwidth for the month.
†In our personal lives, at least. This statement is without prejudice to somebody who hates the technology, but whose work obligates him to use it during his shift.

[Kris_88]

There are real differences. If someone visits a site himself, he actually sees whatever the webmaster has written and wants to share with the world. A bot crawling the site cannot do this. It is a soulless, automated request which copies the page into a database which may or may not ever actually be read. For small servers especially, with limited bandwidth*, a webmaster might wish to reserve his traffic for real people.

A dubious argument. For example, if I'm looking for something specific, I don't read the entire page; I quickly scan it, intuitively finding what I need (sometimes just one post), and then skip the rest. If I find a book that covers a specific point I need, I read that specific point and save the entire book to my hard drive. It's not a given that I'll read it all later. I don't fully read everything I open in my browser, either. If I need to dig a hole, I grab a shovel, but there's no soul in it... Yes, I could dig by hand, but a shovel is faster and easier. So, does that mean I'm a bot?
Even increased traffic is a weak argument these days. Traffic is free almost everywhere, as far as I know. Server load - yes, that's an argument. It simply means limiting the frequency of user requests, whether it's a human or a program configured by a human.

Again, different people have different purposes.

There's no doubt about it. Occasionally, very unusual situations occur.

But I think the strongest objection is that against scraping one’s site ‘to train AI’. Many people, I among them, are categorically opposed to this technology’s existence and abstain completely from it.

This is truly an unusual technology. You just have to be critical of the results AI produces (as you should be of everything you find online). But I wouldn't draw any esoteric or religious implications here. Essentially, AI is a sophisticated search engine masquerading as a sentient being.

[Mæstro]

A dubious argument. For example, if I'm looking for something specific, I don't read the entire page; I quickly scan it, intuitively finding what I need (sometimes just one post), and then skip the rest…

Some publications are not intended for readers who are merely skimming for information. Indeed, I have found site redesigns catering to such readers (I have British statistical offices in mind as my chief concrete example) quite annoying. When we go beyond reference material where nobody expects the textbook to be read straight through, the fact becomes more apparent. If I write a story and post it on my personal website, I want my readers to pay attention to it undistracted, enter the setting I am describing and fall in love with the characters the way I have.

It is true that not everybody who visits a page is certain to give it the attention it is due, but the chances are far greater when someone has clicked on this link thinking ‘I want to visit this page right now’ and has actually got it on the screen than if a spider, worm or some other vermin is accessing the server without anyone directly involved. Automatic requests might receive or bring personal attention, like when the Internet Archive mirrors a site, or a search engine helps somebody discover a page, but it is far less likely, so a website intended really to be read will naturally wish for most of its traffic to be authentic and organic.

Traffic is free almost everywhere…

In my footnote, I shared a recent counter-example. This is especially a problem for small sites who host everything personally. Moreover, I continued to live under rather strict monthly bandwidth caps well after this ceased to be common, until a few years ago. This has left me with the firm belief that the interests of people with low bandwidth ought always to be regarded, as a matter of accessibility.

It simply means limiting the frequency of user requests, whether it's a human or a program configured by a human.

Webmasters might wish for natural persons to make up all, or almost all, their traffic, regulating bot traffic so it stays within prescribed, known bounds. Many do not want their natural traffic to be crowded out by bots.

This is truly an unusual technology. You just have to be critical of the results AI produces (as you should be of everything you find online). But I wouldn't draw any esoteric or religious implications here. Essentially, AI is a sophisticated search engine masquerading as a sentient being.

I do not feel like turning this thread into yet another debate about the merits, or lack thereof, of LLM as a technology. There is much I could say, but it is beside the point. Suffice it to say that many of us have weighed it in the balance, found it wanting and wish nothing to do with it, such that scraping against our will is unauthorised, hence malicious. Many of us here, I among them, have been harmed in many ways by LLM scraper swarms. Among them are the dilution of legitimate search results with plagiarised gibberish, and most importantly for this thread, the resultant spike in JavaScript-based captchas and especially Cloudflare use.

[Gemmaugr]

AI is a sophisticated search engine masquerading as a sentient being.

Just a small correction here. Humans are animals that are aware that they're aware, which is called sapience. Animals that are "just" aware, are sentient. AI is supposed to be masquerading sapience, but it has as of yet to reach even sentience.

[Moonchild]

The mechanics of scraping are also considerably different and exponentially more heavy on the server because many, many copies of the same data is requested because scrapers are not smart enough to not follow links leading to the same data from a different URL.

e.g. on the repo with tagged issues, a scraper will dutifully scrape the same issues under each tag for as many tags are attached to the issue. Same with versions. Same with any other dynamically generated data based on filter criteria. One single page can be crawled hundreds or thousands of times because there are as many different ways to land on it.
Any "AI summary" reducing load by people being satisfied with the (potentially flawed) primer of the target page is insignificant compared to the relentless requesting of every which way one could reach specific content ballooning the total number of requests for that content.

[BenFenner]

A search engine will download your site once