Cloudflare Verification Loop issues

[Moonchild]

{{Gobbledygook about being subject to God}}

I have no fucking clue what you're on about or what you're smoking. Anubis is a tool that challenges web clients. I don't know how google translate operates on the sites they translate but most likely they frame it and make their own calls for content, which can run into bot challenges because it is, in fact, a bot.

My actual question to you, since you were saying "their functionality could be implemented with fewer problems for the user", is that I want to know what insights you have in a way to achieve that. Any concrete ideas. Because I certainly am not married to whatever bot shield if there is a better solution, but there has to be a concrete solution and you're not telling me anything. So if you have some good idea I'd love to hear it. Otherwise, it may be best to just keep your weird theological approaches out of the conversation.

[back2themoon]

Kris_88 is most probably referring to this Anubis which is an interesting connotation but poor little girl Anubis (the software) doesn't deserve this treatment. Cloudflare is way closer to god-like status and behaviour here.

In any case, more alternatives are needed. Never had an issue with Anubis by the way, whether Egyptian god or website-protecting software.

[Kris_88]

Kris_88 is most probably referring to this Anubis

Exactly so.
And this is no coincidence. The project's description on GitHub ( https://github.com/TecharoHQ/anubis ) directly links to the Wikipedia article ( https://en.wikipedia.org/wiki/Weighing_of_souls ), which states: "The Weighing of the Heart would take place in Duat (the Underworld), in which the dead were judged by Anubis".

Quote from the project description along with a link:
"Anubis is a Web AI Firewall Utility that weighs the soul of your connection using one or more challenges in order to protect upstream resources from scraper bots."

So, this is not my invention, but a direct reference from the author of the project.

My actual question to you, since you were saying "their functionality could be implemented with fewer problems for the user", is that I want to know what insights you have in a way to achieve that.

Wait, Moonchild. I said exactly what I said. Yes, I have no doubt that this turnstile could have been implemented in a way that would have created fewer problems for users. In particular, I was referring to the problems I encountered and described above. No, I didn't say I knew a ready-made solution that could be taken and applied to the site. And no, I didn't in any way imply that I was ready to rewrite Anubis to fix its flaws. I simply pointed out flaws that, in my opinion, are the result of the wrong approach of turnstile developers, who put the interests of the turnstile above the interests of both users and websites.

[Moonchild]

I'm aware of the Qemetic belief system and Anubis (Anpu) in that. But what you said implied that websites would have zero choice but to "submit to their gods", i.e. placing Anubis and CF on a theological pedestal that websites MUST be using as if they were under the unwavering rule of such dieties, and that's just not true and implying such an analogy is absurd. If I misread that implication then I apologize.

. No, I didn't say I knew a ready-made solution that could be taken and applied to the site. And no, I didn't in any way imply that I was ready to rewrite Anubis to fix its flaws.

That isn't at all what I asked. I just asked if you had any idea at all how to do it better. Because your conviction was that it could be written better, so you must then have some idea how, right? I never asked for a ready solution, just a reason why you said with certainty that it could be done better because maybe it can help us (and everyone else) if there are good ideas in your head.

[UCyborg]

@Kris_88
When I mentioned Anubis, I simply meant I can open the site in the basic sense. Though I noticed problems with Google Translate with Cloudflare too. They all suck. There's this guy that operates https://board.eclipse.cx/, he didn't put any anti-bot thing on it. I wonder if this is the consequence that it's often so laggy.

[Moonchild]

There's this guy that operates https://board.eclipse.cx/, he didn't put any anti-bot thing on it. I wonder if this is the consequence that it's often so laggy.

Very likely so, and then to consider that that board has maybe 3000 posts on it in total? that's very small in terms of content so scrapers will soon enough "run out" of things to scrape and stop loading the host.

[Kris_88]

I just asked if you had any idea at all how to do it better. Because your conviction was that it could be written better, so you must then have some idea how, right?

You're really pushing me to address this issue... Okay, I looked.
Regarding Google Translate, there's a problem with CORS.

Code: Select all

Warning: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://forum.palemoon.org/.within.website/x/cmd/anubis/static/js/main.mjs?cacheBuster=1.23.1.
(Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

You can add on your server

Code: Select all

Access-Control-Allow-Origin: https://forum-palemoon-org.translate.goog

and then the process will at least start.
But another problem with the worker will arise:

Code: Select all

Security Error: Content at https://forum-palemoon-org.translate.goog/search.php ... 
may not load data from https://forum.palemoon.org/.within.website/x/cmd/anubis/static/js/worker/sha256-purejs.mjs?cacheBuster=1.23.1.

And here we need to look deeper. Something seems to me like Pale Moon has some kind of problem, but I could be wrong...

Actually, I don't like dealing with turnstiles. Some people will do all sorts of crap, and I'll have to fix it. There's a lot of crap in this world, and my life wouldn't be long enough to fix it all.

[Kris_88]

Off-topic:

https://www.humanrightscareers.com/issu ... important/
https://privacyinternational.org/learni ... cy-matters
https://papers.ssrn.com/sol3/papers.cfm ... _id=998565&

Yes, I understand and generally agree with the articles you linked to.

Of course, I have personal information, and it's highly undesirable for it to fall into the wrong hands. For example, this data could be used by scammers to harm me, and of course I don't want that. And I protect such sensitive information, meaning I don't distribute it, post it on forums, send it by email, don't use online translators for it, etc. I also don't run unknown applications on my computer or applications that I'm even slightly unsure about, assuming my personal information is stored or accessible on the computer. And I try not to store sensitive information directly on the computer I work on.

In other words, I protect my personal information, and not just from Google, but from everyone.

As for everything else, I'm not too worried. Well, let search engines collect my queries... There's nothing truly secret or sensitive in them.

[frostknight]

Oh, what data do I have...
The fact that I visit this site and even write something sometimes? That's no secret... And I don't do anything illegal, so I don't have any special secrets. Let them collect data, it doesn't bother me...

This is how fascists want people to think.

They will use everything against people if they feel like it.

[Gemmaugr]

So, is it safe to assume that most (all?) of us are currently in a good, working state where CF is concerned?

Yup. Works good. It's sad that you have to click an extra button, which is not how it used to be, but it does work. I tested just now with https://dbsoft.org/whitestar.php

[back2themoon]

It's sad that you have to click an extra button...

Perhaps a userscript could take care of that?

[BenFenner]

I want to know what insights you have in a way to achieve that. Any concrete ideas.

https://forum.palemoon.org/viewtopic.ph ... 47#p271047
Copy what I've done. I'll send updates if/when I make them as needed. So far this has completely solved the problem, and rarely needs updating. It was a night and day difference. A proper silver bullet for my forum (12k+ users, 75k+ threads, 1M+ posts).

robots.txt (Obviously not needed, but I like to give them one chance to do the right thing.)

Code: Select all

User-agent: *
Disallow: /.htaccess

User-agent: AhrefsBot
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /
User-agent: barkrowler
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: Scrapy
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: serpstatbot
Disallow: /
User-agent: trendictionbot
Disallow: /


User-agent: AI2Bot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: DuckAssistBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Google-CloudVertexBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Kangaroo Bot
Disallow: /
User-agent: Meta-ExternalAds
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: Meta-WebIndexer
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: PanguBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
User-agent: YouBot
Disallow: /

.htaccess (change the two /http_status_codes/403_forbidden.php lines as appropriate)

Code: Select all

RewriteEngine On
RewriteBase /

# Block other unwanted/annoying/useless scrapers.
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Amzn-SearchBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Barkrowler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} BLEXBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DotBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Scrapy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} serpstatbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} trendictionbot [NC]
RewriteCond %{REQUEST_URI} !=/robots.txt
RewriteCond %{REQUEST_URI} !=/http_status_codes/403_forbidden.php
RewriteRule ^(.*) - [F]


# Block common scrapers that use our data to feed/train their AI tools.
# Based on (but since augmented): https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website
RewriteCond %{HTTP_USER_AGENT} AI2Bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Amazonbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Applebot-Extended [NC,OR]
RewriteCond %{HTTP_USER_AGENT} anthropic-ai [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ChatGPT [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ChatGPT-User [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Claude-SearchBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Claude-Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cohere-ai [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cohere-training-data-crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Diffbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} DuckAssistBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} FacebookBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Google-Extended [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Google-CloudVertexBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ImagesiftBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Kangaroo Bot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Meta-ExternalAds [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Meta-ExternalAgent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Meta-ExternalFetcher [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Meta-WebIndexer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Omgilibot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Omgili [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PanguBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Timpibot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Webzio-Extended [NC,OR]
RewriteCond %{HTTP_USER_AGENT} YouBot [NC]
RewriteCond %{REQUEST_URI} !=/robots.txt
RewriteCond %{REQUEST_URI} !=/http_status_codes/403_forbidden.php
RewriteRule ^(.*) - [F]

If not using Apache, I'm sure I can help translate it to Nginx, or whatever this forum uses...

[Moonchild]

robots.txt

AI scrapers don't honor robots.txt at all.

UA filtering is pointless as they go out of their way to use a smattering of Chrome UAs when scraping, and do not identify as bots in the UA at all.

Example of a scraper UA:

Code: Select all

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36

[BenFenner]

AI scrapers don't honor robots.txt at all.

I mean they should... But yeah, no shit most don't. I said as much in my post.

UA filtering is pointless as they go out of their way to use a smattering of Chrome UAs when scraping, and do not identify as bots in the UA at all.

Are we on different Internets? Or are you just making assumptions? Have you tried? Like I said it was night and day for my forum. The bot traffic problem went below the leve of noise. Please reconsider.

I think you have missidentified the source of the bulk of problematic bot traffic. The sources are "legitimate" outfits using predictable UAs. Tell them to pound sand and your woes will go away.

(The sources being "legit" is also why I give them a chance to do the right thing with robots.txt.)

[Moonchild]

Are we on different Internets? Or are you just making assumptions? Have you tried?

Yes, this is from experience looking at my nginx logs. I guess we are on different internets if all the traffic you see dutifully identifies itself as bots.

No asking nicely with robots.txt will make them go away. No filtering on *bot UAs will make a difference because, as said, I clearly see the bot traffic and they are using Chrome/Safari UAs, often faking Mobile as well. How do I know they are bots? Because they come from tightly-packed IP clusters in datacenters (where you'd expect those requests to come from singular IPs over a keep-alive http), or use distributed networks but those are easy to spot as well because bots do not show predictable "normal" browsing patterns, i.e. you will see requests for forum topics with no surrounding traffic at all for navigation or styling, and they often come with session ids in every request (normally handled by cookies) that can even be randomized. Another giveaway is escaped URLs, or even worse half-escaped URLs (when fully-escaped ones obviously get filtered).
I just got tired of cat&mouse games getting almost daily reports from the host node that there was excessive load and seeing it being yet another subtle variation bypassing a UA filter, so I installed anubis. And it's been blissfully calm ever since and I only on occasion need to adjust the bot policies in it.

So yeah, a little less doubt about my sincerity would be appreciated.

[Goodydino]

Yup. Works good. It's sad that you have to click an extra button, which is not how it used to be, but it does work. I tested just now with https://dbsoft.org/whitestar.php

That page crashes Pale Moon for me.

[Night Wing]

That page crashes Pale Moon for me.

Does not crash for me. You can see what I am using in my signature.

[Sessh]

I was able to get into the dbsoft.org link with no issues.

[back2themoon]

It's sad that you have to click an extra button...

Perhaps a userscript could take care of that?

Seems difficult, if not impossible (?) for a userscript to handle this.

Easier solution: <2xTAB> + <SPACE> which I guess can be semi-automated in several ways.

I also now noticed some checks (not all) let me right in with no mouse click.

[Sessh]

Just saw this and figured it was relevant. I wonder if CF is gonna make a bunch of changes soon to mess things up again for us... but they're not having a good time if bots can now solve CF challenges.

Clipboard01.jpg