CloudFlare: bot detection speculations

[Kris_88]

Moderator note: split off to keep status topic on-point.

So what's not clear here? Of course, it would be convenient and cheap to recognize bots using only a script that runs on the client side and does not require user intervention to solve the captcha in its classical sense. The problem is that such a script is impossible.
Therefore, the task of recognizing bots is replaced by the task of recognizing widespread browsers and checking for the absence of known add-ons for automation. And even such a limited task is not simple.
That's why it turns out this way... The problem is not in the competence of the developers, but in the fact that the original problem has no solution.

[Nuck-TH]

They somehow managed to have workable solution for years, but now suddenly it needs newest JS shinies and fragile fingerprinting to work, yet falls to stupid trick of presenting it cookies from browser that passed check. Something doesn't add up here. And generally easily bypassed. For example search github - there are tons of scripts,bots,etc that bypass CF checks easily and currently work.

Off-topic:
And, you know, big fat corp that is bathing in money from both their clients and selling data(and don't start on that they don't) don't need your defense. Defending corps or being fanboy gives you absolutely nothing, but it does encourage further anti user behavior.

[Kris_88]

They somehow managed to have workable solution for years, but now suddenly it needs newest JS shinies and fragile fingerprinting to work,

It all fits together. Probably some bots that are built on forks of famous browsers will not pass the test for the latest JS capabilities. Simply because forks become obsolete if you don't work on them.

yet falls to stupid trick of presenting it cookies from browser that passed check.

What else can you do if you don't want to store any client information on the server side?

And no, I'm not defending them. It's just obvious to me that they've chosen a cheap but simplistic solution to the bot detection problem that gives a lot of false positives.

[Moonchild]

Probably some bots that are built on forks of famous browsers will not pass the test for the latest JS capabilities. Simply because forks become obsolete if you don't work on them.

And at the same time mainstream browsers are happy to implement WebDriver and other automation tools that make bots indistinguishable from a client-detection PoV. There won't even be a need to fork, just use the latest version, and hook up selenium or marionette to do your bot-things with it.

As I stated before, what CF should be doing for a long-term solution is to do actual behavioural analysis on the traffic they so happily absorb, and focus on that, not on which client signature they happen to get. That is certainly something I'll bring up in the "browser development program" that has been proposed.

[Kris_88]

And at the same time mainstream browsers are happy to implement WebDriver and other automation tools that make bots indistinguishable from a client-detection PoV.

These technologies are actually outside my area of ​​interest, but a quick Google search shows that webdriver can be detected in a variety of ways.

, what CF should be doing for a long-term solution is to do actual behavioural analysis on the traffic they so happily absorb

And this requires storing some information about the client on the server side. Moreover, if there are many servers, then this information should be shared among them... Do you think they don't know about this solution?

[Moonchild]

but a quick Google search shows that webdriver can be detected in a variety of ways.

All of which can be spoofed, AFAIK. If not natively then with a minimal rebuild short-circuiting various tattle-tales in the code.

this requires storing some information about the client on the server side

...and you think they don't already do that...? Sorry but that's very naive.

if there are many servers, then this information should be shared among them... Do you think they don't know about this solution?

Sharing would not necessarily be needed; bots do not tend to distribute a single attack source over multiple PoPs of CF which is determined by their own DNS cast anyway.
Speculation: I think they do know about this solution, and I think they don't want to do this because it would mean more cost and thus smaller margins.

[Michaell]

I'd like to ask a broader question and I'm putting it here not to derail the thread that the CF rep is participating in.

WHY? Why is it considered ethical (legal?) for CloudFlare to be doing this at all? When I first learned several years ago that CF (I thought it was CloudFare when they started but maybe I misread it repeatedly although that fit better IMO) was intercepting encrypted connections, I felt almost like calling the police. I recall there was something sort of similar with ISPs earlier and I thought the result was they were not permitted to do this. And I'm thinking specifically of the times when I get CF errors from sites that have no indication they are going through CF until a CF error page comes up. If https encryption protocols are broken by servers we don't even know we're connecting to, then it feels like we've been played for suckers all this time.

[Kris_88]

Why is it considered ethical (legal?) for CloudFlare to be doing this at all?

As far as I understand, everything is legal and ethical. Cloudflare does its job on behalf of the site owner. Just as the site owner may not have his own server, but use hosting services, and then the hoster has access to all the data. Cloudflare does not integrate between the user and the site on its own initiative. This happens on the initiative and consent of the site owner.

[jobbautista9]

If a bank is using a third-party CDN for TLS (which they probably shouldn't be) then the bank is to blame for not ensuring end-to-end encryption between you and their servers, IMHO. So I agree with Kris here.

I can understand sysadmins lazily enabling "bot attack" mode and forgetting about it (they still have some blame, but Cloudflare also deserves blame for marketing it as foolproof protection), but it's inexcusable for sysadmins of sensitive sites to not think about the security implications of their website design, which includes how content is delivered.

[Moonchild]

WHY? Why is it considered ethical (legal?) for CloudFlare to be doing this at all?

It is both ethical and legal for CloudFlare to do this. They are providing a service to their clients, and how they are implementing this service is up to them.
That doesn't mean they can do whatever they want in that implementation, though, if their practices create losses for others through tort (legal issue) or assume it is without consequence for their customers, especially if they cover a large portion of the exposed public internet.

There is absolutely blame to be cast towards CloudFlare for this happening, but it doesn't by itself make a legal or ethical issue. It's only within the context of their coverage and scale that this becomes an issue; I likened it before to the analogies of CloudFlare at its scale becoming more akin to public utility than a simple B2B understanding, which puts the bar higher for their business operations than they might have held to in the past.

This happens on the initiative and consent of the site owner.

Not as black-and-white, when the issue is marketed as desirable and clients are advised heavily to make use of it. The "initiative" part comes in question in that case.