CloudFlare discussion thread

[mstremante]

Moderator note: Split off to focus this topic on discussion between community devs and CloudFlare. Please keep this focused and on-topic!

Hi - Michael here. Thought I'd jump into the forum directly.

With this knowledge as well as my feedback how to easily detect Pale Moon's user-agent

I'm sorry but I never received information on how to consistently identify Pale Moon browsers. I also did not receive any notification bar one from the mailing list you set up so have not seen any questions from the group. I suggest we also keep the discussion here so it’s easier to track for everyone. I will send an email to the mailing list shortly to confirm it's working well.

Whitelisting is easy, is an NDA needed for that?

As I said in the email, but missed in the summary above, I myself stated that I hope we won't need it. The team sent the generic NDA as writing anything custom, although doable, does require more time. For now we still don’t need it.

They've gone silent and/or time stalling because they achieved their goal

Quite the contrary and this has never been our goal. There is also no anticompetitive angle we are seeking. We don’t participate in browser related engagements to date. We don't have logic in our systems that specifically target the major browser vendors. We rely on well known browser standards to implement our checks.

With the team we agreed we can be more specific with the issues identified as it does not pose a security risk. Pale Moon lacks support for these two functions:

• canvas fillText()

• DomRect.toJSON()

We have a PR ready to remove checks against these two functions for Pale Moon. We still need an authoritative answer on how to identify Pale Moon clients, otherwise we can simply check against the standard Pale Moon user agent. Please let us know.

Pale Moon does need to support CSPs as well (including in web workers). Specifically these directives:

• script-src 'unsafe-eval';

• worker-src blob:;

• frame-src 'self' blob:;

• child-src 'self' blob:;

At this moment, it's not possible for us to remove checks for these for Pale Moon. We continue to explore options, but if Pale Moon implements these, we can take care of the lack of JS function support listed above.

I will follow up with the above information in the mailing list as well.

Michael

[Kris_88]

Pale Moon does need to support CSPs as well (including in web workers). Specifically these directives:

• script-src 'unsafe-eval';

• worker-src blob:;

• frame-src 'self' blob:;

• child-src 'self' blob:;

As for CSP, it's probably not these instructions that are the problem.
The "EvalError : call to eval() blocked by CSP" message аnd a similar message about creating a worker from a blob, that confused me are probably caused by executing the script in an iframe detached from the DOM. That's a guess. I don't know the exact reason, but it seems like there are multiple copies of the script running at the same time and the old copy is generating CSP errors.
It's quite difficult to understand your scripts...

[andyprough]

The team sent the generic NDA as writing anything custom, although doable, does require more time. For now we still don’t need it.

That's good to hear Michael, I hope we can continue without an NDA,which would have to be narrowly tailored to specific technical questions and would have to meet legal requirements for multiple jurisdictions. For now, I would assume that this discussion thread (or something similar in the developer area of the forum) will be useful.

[Moonchild]

I'm sorry but I never received information on how to consistently identify Pale Moon browsers.

I sent it to the mailing list (which e-mails each member individually, and your e-mail address is clearly the same as what was used individually) on the 11th of March 00:26 CET. If you received any notification from the mailing list, as you say, then you should also have received this, as well as the other discussions that happened; but I'll repeat it here, just in case, since you indicated to want to use the forum instead anyway.

> Q: Is there a way to consistently identify the Pale Moon browser?
There is, and it's outlined in viewtopic.php?f=24&t=28233 as a general pointer, if you are using user-agent strings. In short, official builds of Pale Moon will always include a substring identifier "PaleMoon/nn.nn.nn" where nn.nn.nn is the browser version as well as "Goanna/*" where * is either the build ID or rendering engine version depending on which compatibility mode is chosen in the browser's options (should be common for all browsers building on UXP). It is, however, important to note that other browsers on the same platform (particularly Basilisk, the dev of which is also on this list) uses a different browser identifier instead of Pale Moon and the same would be true for other rebranded browsers and also for unbranded ones.

There were no bounces or delivery issues to your e-mail address from the mailing list. If there had been, I would have immediately noticed.

About the NDA:

I myself stated that I hope we won't need it. The team sent the generic NDA as writing anything custom, although doable, does require more time. For now we still don’t need it.

I'm sorry but if it's not needed, why was it sent to me for my signature? It was clearly intended to be a binding agreement if I went through the document signing process when I received it. It was clearly sent to me for signing. There was zero indication that this was a draft, merely a proposal, or anything else non-final. The document received was already signed by CloudFlare just needing my signature to complete.
You do understand, I hope, how this looks from our end, right?

Image1.png

We don’t participate in browser related engagements to date.

While that may be true, as stated back in 2022 and in our previous discussion here, it is imperative that you do engage with browsers as they are the very clients you need to work with for accurate block/pass behaviour of your "bot detection". I shouldn't have to remind you of this as that would literally be the core description of your bot detection dev team.

We don't have logic in our systems that specifically target the major browser vendors. We rely on well known browser standards to implement our checks.

I'm sorry but it seems you do. Community members have tested various things, among which using a "supported" browser with a Pale Moon user-agent string and got promptly blocked. We've also seen varying behaviours of the captcha based on user-agents, so it's absolutely clear that you aren't solely using JS features for your bot detection. The "well-known" browser standards are also not well-known to anyone outside of CF (in part because the scripting is highly obfuscated; presumably to make reverse-engineering harder). In addition, you are also relying on very complex and convoluted checks that are exquisitely implementation-dependent (underlined by the fact something as simple as hardware acceleration being unavailable in otherwise "supported" browser engines will fail the bot check). Ultimately it all combines to a checking suite that is implementation-dependent to such a degree that it would, necessarily, have to be catered to each individual browser implementation and version, as seems to be the case. While I understand, from that point of view, that you short-list what your challenges "support", the problem is, as I have already explained before, that you are effectively gatekeeping websites that have no issue being rendered in all affected browsers by having this arbitrary, very narrow access gate.

CSP:
As a critical overarching note, relying on CSP to do bot-detection is IMHO very bad practice, as any bot could simply not respond to CSP headers and allow whichever response you're attempting to specifically restrict. It is much more likely that clients responding to CSP are legitimate browsers rather than bots.

script-src 'unsafe-eval';

We do support this. However, there may be implementation-specific differences you rely on. Your dev team will have to provide specific details about this.

worker-src blob:;
frame-src 'self' blob:;
child-src 'self' blob:;

Note: child-src has been officially deprecated by the W3C in 2022.

At this moment, it's not possible for us to remove checks for these for Pale Moon. We continue to explore options, but if Pale Moon implements these, we can take care of the lack of JS function support listed above.

Pardon my French, but it is absolutely bloody possible for you to remove checks for these for Pale Moon. You just are deciding not to. It worked before Jan 31st, you know which checks are the problem, so please find a way to make it work.
We can look into implementing blob: for workers (which is what seems to be missing) but that will not be a short-term or immediate thing. I do hope you can find a short-term solution to give us the necessary time to work on our CSP implementation. Note that the CSP spec is very complex and wrought with spec changes, implementation differences and this particular use of it really falls outside of the design scope for CSP (which is primarily meant as an active XSS protection measure!). We have always approached CSP from the XSS attack surface stand-point and have often been more strict in what we allow than mainstream implementations like Blink. Once again, it seems quite counter what a bot would do...

With the team we agreed we can be more specific with the issues identified as it does not pose a security risk. Pale Moon lacks support for these two functions:

canvas fillText()
DomRect.toJSON()

Once again this can be prioritized on our end but it won't be an immediate thing, and in the meanwhile our inability to access sites is harmful to us.
It does seem that especially canvas fillText() would be used to rely on specific features of graphic representations that fall immediately in the realm of browser fingerprinting. Please be aware that this is in conflict with people valuing browser privacy who may explicitly disable canvas features or poison the results and should not be considered a reliable check for bots.

As I already hinted at before I feel this is a much too convoluted and complex way of checking, which instead should be handled by investigating the actual traffic instead of offloading it to a very narrow selection of browser features that must be matched perfectly to pass. IMHO the focus should be on legitimacy of traffic, not client, but if you do want to check legitimacy of clients, then your legitimacy check should include every and all actively developed browsers, not just "whichever has the biggest market share".

[tellu-white]

We still need an authoritative answer on how to identify Pale Moon clients, otherwise we can simply check against the standard Pale Moon user agent.

Given how much time has passed and how many obstacles have been overcome to get to this stage of communication, we may wonder if you need "an authoritative answer on how to identify Pale Moon clients" to get the problem solved or to more easily block any possible bypassing of your checks made by Pale Moon users?!

No offense, but given the circumstances, this is a legitimate question.

[Basilisk-Dev]

We have a PR ready to remove checks against these two functions for Pale Moon. We still need an authoritative answer on how to identify Pale Moon clients, otherwise we can simply check against the standard Pale Moon user agent. Please let us know.

To be clear, this issue affects the Basilisk browser as well. You guys broke things for several browsers with the change being discussed here, not just Pale Moon. I'm not sure why your company is so adamant on blocking legitimate traffic from UXP based browsers. This has happened at least 3 times in the past, and this is the only time we have been able to get an employee of Cloudflare to acknowledge our existence at all.

I think it might be more beneficial to find a way to identify that a browser is built upon the UXP platform rather than identifying that it's Pale Moon specifically. Someone building a fork of Pale Moon or building with unofficial branding might have a different user agent; or in the case of Basilisk, while it uses the same rendering engine as Pale Moon it is not Pale Moon and does not use a Pale Moon user agent. User Agent scanning is not a good way to verify if traffic is legitimate or not as it is extremely easy to spoof a user agent.

Also, as already discussed via the mailing list I am willing to sign an NDA if required although I would prefer not to have to sign one.

[BenFenner]

We don't have logic in our systems that specifically target the major browser vendors.

Please, don't insult our intelligence. That's a fun claim to make, but ultimately meaningless. The end result of specifically targeting, or surreptitiously targeting, or systemically targeting, or targeting by proxy, or targeting by happenstance, major browser vendors is the same.

On that topic, do you use any browsers at all during your test regime (formal/informal, automated/manual)? If so, you target those whether you like it or not. This will be the 3rd time in recent memory we ask that you add Pale Moon and Basilisk to that process.

We rely on well known browser standards to implement our checks.

No, you don't. You won't find a more standards-compliant browser than Pale Moon.

[biopsin]

I'm adding this post as CF breaks on disabling npapi support, which was technically depcricated in 2015.

Code: Select all

https://forum.palemoon.org/viewtopic.php?f=5&t=31097&hilit=npapi

[Kris_88]

• canvas fillText()

• DomRect.toJSON()

Indeed,
CanvasRenderingContext2D.fillText()

And probably also:
window.navigator.deviceMemory
Window.structuredClone([ BigInt ])
SVGGElement.getBBox()

[Moonchild]

window.navigator.deviceMemory

There is no reason to expose system properties like this to web content. Other than fingerprinting or telemetry it has no practical purpose.

[Kris_88]

There is no reason to expose system properties like this to web content.

No problem.
It just seems like their script uses it...

[Moonchild]

It just seems like their script uses it...

It shouldn't.

[BenFenner]

Moonchild, it seems Michael from CloudFlare would prefer a non-user-agent method of identifying Pale Moon. Likely a much less easily proofed identifier? (Which of course makes total sense to want in some contexts.)

Is that not readily available?
Is it perhaps a convoluted fingerprint only knowable if the fingerprints of other browsers are also known?
Is it perhaps technically impossible, or against the spirit of the project?

(I will refrain from going off on a tangent about CloudFlalre wanting to identify Pale Moon when they proudly intimate they don't implement per-browser code.)

[Moonchild]

Likely a much less easily proofed identifier? (Which of course makes total sense to want in some contexts.)

Is that not readily available?

It is absolutely doable, but it does mean that they need to actually enumerate our feature set and test their scripting against Pale Moon. Since we don't know what CloudFlare tests against exactly and they don't want to disclose that kind of information to the public, it is ultimately something CloudFlare must do themselves. They can use the cash they get from their enterprise customers to do their job.

our Enterprise plan, usually priced at several thousand dollars per month.

If they want a quick overview and enumeration of the javascript/ES features we support, they can check the ES2016+ compat table from an instance of Pale Moon or other UXP browser.

[mstremante]

All,

(lots of replies, so excuse me if I miss some comments - I also posted early by mistake, so pardon the broken post)

As for CSP, it's probably not these instructions that are the problem. The "EvalError : call to eval() blocked by CSP" message аnd a similar ... It's quite difficult to understand your scripts…

The error is intentional and part of our checks, so the behaviour here is by design.

There is, and it's outlined in viewtopic.php?f=24&t=28233 as a general pointer

Thank you very much.

I'm sorry but if it's not needed, why was it sent to me for my signature? ... You do understand, I hope, how this looks from our end, right?

We sent the NDAs from our legal software portal. There was no other intention than just sharing the NDA doc with you for review. We were not expecting or requesting a signature in any way.

it is imperative that you do engage with browsers as they are the very clients you need to work with for accurate block/pass behaviour of your "bot detection".

We are looking at setting up an official browser developer program. We are planning to have a draft soon.

I'm sorry but it seems you do. ... you are effectively gatekeeping websites that have no issue being rendered in all affected browsers by having this arbitrary, very narrow access gate.

Note that we do not issue any challenges out of the box. These are customer configurations from their WAF/Bot Management dashboards normally in response to a bot problem. We are always trying to improve visibility around challenge solve rates to inform our users about the impact of their rules.

As a critical overarching note, relying on CSP to do bot-detection is IMHO very bad practice, as any bot could simply not respond to CSP headers and allow whichever response you're attempting to specifically restrict. It is much more likely that clients responding to CSP are legitimate browsers rather than bots.

You are correct. We also try to identify legitimate browsers to reduce the chances of blocking humans. For that, checking for CSP support is good.

We do support this. However, there may be implementation-specific differences you rely on. Your dev team will have to provide specific details about this.

The handling of CSPs in Pale Moon seems to be quite different from other browsers. Some of the forum members have started some investigations on this front.

Note: child-src has been officially deprecated by the W3C in 2022.

We support both frame-src or worker-src, but expect child-src to be handled correctly as an expected fallback.

I think it might be more beneficial to find a way to identify that a browser is running on the UXP platform rather than identifying that it's Pale Moon, as someone building a fork of Pale Moon or building with unofficial branding might have a different user agent.

We agree with this sentiment. Is there a way to identify both Pale Moon and Basilisk? That is not, as another user noted, just UA based? If not, we shall go with UA only.

On that topic, do you use any browsers at all during your test regime (formal/informal, automated/manual)? If so, you target those whether you like it or not. This will be the 3rd time in recent memory we ask that you add Pale Moon and Basilisk to that process.

We are looking at options to include Pale Moon in an automated fashion.

Pardon my French, ...Once again, it seems quite counter what a bot would do…

In the interest of moving forward we are open to removing CSP checks for PM as long as you can also come forward and commit to a timeline for implementing the relevant APIs? We would then reintroduce those checks once the agreed timeline has expired. Note that we may need to instrument anonymised aggregate traffic analytics for the PM UA as it is likely that bot developers would start leveraging this in nefarious ways. If that happens we can discuss next steps. As another user notes, it is odd that we are suggesting an exception for PM, when we do not have other browser specific code in place. Again, this is for us to find a happy resolution.

Michael

[Bilbo47]

I'm not comfortable with the idea of writing in exceptions for Pale Moon, nor for all UXP browsers, nor for any particular browsers nor rendering engines. That whole approach would seen to make the apparent gatekeeping problem worse overall, not better. What's needed is a basic policy intent to support all browsers that are driven by humans. We can't respond to only the noisiest groups of users.

We already know that bot detection can measure time between elements of UI events, even when such automated events are programmed with random time delays. Other than that I don't know enough about the details to comment, but wanted to toss in my two cents about the big picture. Thank you for discussing

[BenFenner]

What's needed is a basic policy intent to support all browsers that are driven by humans.

In theory that is found here?
https://developers.cloudflare.com/ssl/r ... patibility

In practice...

[Moonchild]

We sent the NDAs from our legal software portal. There was no other intention than just sharing the NDA doc with you for review. We were not expecting or requesting a signature in any way.

Please don't insult my intelligence.

We have prepared and attached "Non-Disclosure Agreement (Cloudflare Paper) (Moonchild Productions and Cloudflare)" for your signature. Please review and sign the document at your earliest convenience.

That is a very clear request to review and sign the NDA.

We are looking at setting up an official browser developer program. We are planning to have a draft soon.

I absolutely welcome this effort even if late. But that still doesn't address the much more immediate problem since this would be a more structureal, long-term program to set up to (finally) streamline integration of web clients. The more immediate problem is the fact that we are now over 1.5 months into the situation where many thousands of websites are inaccessible by Pale Moon and other browser users and on top, in Pale Moon's case, your scripting causes severe impact on our runtime due to hangs inside the script.

Note that we do not issue any challenges out of the box.

I am aware of this. But at the same time you absolutely lean heavily on this functionality in your marketing, so of course many of your clients will be enabling what they perceive as an easy way to "keep bad traffic at bay". So the gatekeeping is a direct result of your marketing being successful.

In the interest of moving forward we are open to removing CSP checks for PM as long as you can also come forward and commit to a timeline for implementing the relevant APIs? We would then reintroduce those checks once the agreed timeline has expired.

I feel this is a topsy-turvy demand. How long this will take depends on how easily we can port code from Mozilla. We may have to implement from scratch (preferably without breaking already-existing code). It's not possible to provide a timeline until we already have a good handle on the problem and are already in the process of getting this implemented, which would be pretty far into the process. I do not feel it is fair or appropriate that CloudFlare uses this as a conditional to "allow us through the gate". That isn't how this is supposed to work in a net-neutral environment.
With how you particularly agree with CSP being a poor mechanism to detect bots earlier on, it makes no sense that you are still wanting to hard-block us on it, anyway. While I have no issue committing to implementing this with priority, it is not possible to give a specific timeline or deadline for it at this time. No issue communicating immediately to you when it has been implemented, so you can put your preferred method back in place, but it's not possible to give you a timeline right now ahead of time. Software development is never that predictable.

In addition, you are still blocking a whole range of other browsers that are not specifically having this particular technical hurdle to clear, which makes:

We also try to identify legitimate browsers to reduce the chances of blocking humans. For that, checking for CSP support is good.

...feel particularly false.

[Pelican]

The handling of CSPs in Pale Moon seems to be quite different from other browsers.

Content-Security-Policy (CSP) is applied from websites in page headers, right?

If CF is testing the web browser to see how it complies to CSP, I am interested in knowing just how it does that because I am wondering if such tests can be reliable.

[BenFenner]

In the interest of moving forward we are open to removing CSP checks for PM as long as you can also come forward and commit to a timeline for implementing the relevant APIs?

I have to say from a browser user's point of view this is an extremely confusing take (to put it nicely). From the point of view of a browser user, the browser is working as intended. It is CloudFlare's responsibility to live up to their marketing hype and stop DOS-ing the very real, legitimate, human users of these browsers. It is not the user's nor browser dev's fault you oversold your technical capabilities.

You also have to realize what you're doing is effectively trying to negotiate with browser devs while actively waging a denial-of-service attack against their users. How about you stop the DOS attack first and then come back to the table with negotiations? Show a modicum of empathy?