Repo server now Anubis-protected

[Moonchild]

Because of the stupidity that is AI scraping (of exponentially many copies of the exact data because scrapers are really not intelligent) I've now installed the Anubis traffic filter in front of the repo server. This will task the browser with a javascript challenge before first access; just have some patience. Subsequent accesses will be much faster after the browser has passed the required Proof of Work challenge.

If you run into issues with this, let me know.

[Tronald Dump]

Wouldn't simple captcha suffice? Or is everyone using Death By Captcha now?

[Moonchild]

You can't incorporate a captcha on every page load.

[Massacre]

Maybe some fail2ban action will suffice? If they are not changing IP during one scrapping session...

[vannilla]

Maybe some fail2ban action will suffice? If they are not changing IP during one scrapping session...

Linking every piece of literature would take too long, just know that every "traditional" solution you are thinking of has been tried already (not by Moonchild, in general by every web service) and they do not work; Anubis is the one that seems to work reasonably well especially under constrained resources.

[Moonchild]

I've manually mitigated the worst offenders the past couple of months, and they use all the tricks in the book to make traditional banning pointless: large swathes of IP pools from cloud providers, varying useragents, etc. you can have 100+ scraping requests causing massive server load without a single repeat IP, for example. Rate limiting is marginally effective only, requiring broad scope checking, and I've had to outright ban certain geographical regions to be able to deal with the worst (which I'd prefer not to do) so no, there isn't a "traditional" way to handle this because of the scale and scope of the bad actors.

[frostknight]

Anubis is not a huge problem. I deal with it on invidio instances all the time. Its a minor annoyance. Not like recaptcha and hcaptcha and others.

Recaptcha is pure evil btw.

Anubis? is actually decent.

[Mæstro]

I've had to outright ban certain geographical regions to be able to deal with the worst (which I'd prefer not to do)

Which places have been banned? A list would be helpful for anyone who happens to live in such a country and is confused to find that he is being blocked.

[Moonchild]

Primarily China (for bot hammering), India and Pakistan (for fake accounts trying to SEO the shit out of gitea).
They get an explanatory page though.
Once I'm confident Anubis is doing a good job, I'll probably remove it again (as a test first and if no issue, permanently)

[Tronald Dump]

Why would you need a captcha every page load? Once a day should be enough. For scraping and similar activities you collect a huge list of proxies and simply keep switching as the proxies get blocked. If there's a global captcha, then you have only 2 choices, either complete it by yourself (scrapers won't bother) or use DBC. Unless it's actually worth it, you usually don't want to pay for DBC.

[Moonchild]

Not sure what you're blathering on about. This solution in place now works -- doesn't require people to do anything except be a little patient the first time they visit, and is pretty hands-off otherwise. There's no way to incorporate a captcha unless you're happy to hit it with an API request each page load (which may or may not trigger a challenge), which in the case of providers like Google for recaptcha is just feeding them user profiling on a silver platter (and recaptcha isn't free if large volume, either).

[hexydec]

I have a bot that is scraping one of your RSS feeds (its a good bot, only once per day). Anubis is now blocking this. Worth allowing correctly announced bots to RSS feeds?

[Moonchild]

I can adjust the policy for it, but I need to know exactly what you're requesting. What is the useragent and what RSS URLs are you requesting?

[q160765803]

But the GeoBlock is still here?

[hexydec]

I can adjust the policy for it, but I need to know exactly what you're requesting. What is the useragent and what RSS URLs are you requesting?

Hi, the user agent is:

Code: Select all

Mozilla/5.0 (compatible; Hexydec Browser Versions Bot/1.0; +https://github.com/hexydec/versions/)

The RSS feed is https://repo.palemoon.org/MoonchildProductions/Pale-Moon/releases.rss.

Thanks.

[Moonchild]

See if it works for you now.

[gabrgv]

I’ve had Basilisk closing itself two times when facing the Anubis challange in the repo server. Unfortunately I can’t provide actually useful data. It may be due to the way I handle cookies: I sanitize every time I close the browser, which is often.

One thing I notice is that this is the Anubis challange that takes the longest to verify. Also I remember sometimes reading that “verification is taking longer than expected” and “difficulty: 4”, but it passes nonetheless.

[Moonchild]

One thing I notice is that this is the Anubis challange that takes the longest to verify.

How long the challenge takes depends on multiple factors; the challenge is also not fixed-time, but proof-of-work, which means it can be quick or it can be slow depending on randomness involved.
I do recommend for your own sanity to not wipe cookies for the repo server so it doesn't trigger every time. As you should be well-aware, we don't track users and we don't serve ads, so all cookies stored for us are by definition all functional cookies.

[hexydec]

See if it works for you now.

Apologies for the delay, yes all working now thanks.

[Moonchild]

Apologies for the delay, yes all working now thanks.

Great. Thanks for the feedback.