Palemoon crashes when accessing Devuan site

[frostknight]

The post doesn't provide anything I can use. it's just vague observations "somewhere between 116 and 120". Mozilla's constant code churn will have thousands of code commits in such a time span.

Yeah, I don't know what he meant either. Maybe the link I gave you would help... I don't know.

[Kris_88]

Generally speaking, nothing surprising...
Anubis creates a worker (and if it can, then several at the same time) and in this worker in an infinite loop calculates SHA-256 hashes using the crypto-API. And naturally, all this is built on promises. And of course, it takes up 100% of the CPU for a long time. And it doesn't care that the computer has other tasks and the browser has other tabs. In short, sorry, but shit is shit.

At least, if I open only one tab with Anubis and do not touch the browser with my hands, then after 2 minutes the test ends and the site opens. But if I somehow interfere with the browser and force it to do something else, then it crashes. And in this case, anything can be the cause - either the GC does not have time to free up memory, or fragmentation grows, or something like that. This is not a normal operating mode for a browser.

[Massacre]

Anubis is definitely crashing Pale Moon (32-bit) when alt-tabbing out of it. Please add mitigation ASAP. You can test it on your own repo.

[Moonchild]

Please add mitigation ASAP.

I don't think it's possible to mitigate this on our end. If you run out of memory, then you run out of memory - end of story. At that point, the only 2 things the browser can do is break or exit. We choose to exit in a safe way.
Anubis does state this is supposed to be a temporary solution - but in the interim this should be handled better. I'll open an issue on their repo and see what they say.

[q160765803]

Please add mitigation ASAP.

I don't think it's possible to mitigate this on our end. If you run out of memory, then you run out of memory - end of story. At that point, the only 2 things the browser can do is break or exit. We choose to exit in a safe way.
Anubis does state this is supposed to be a temporary solution - but in the interim this should be handled better. I'll open an issue on their repo and see what they say.

I wonder if we can't limit how many memory can be allocated by user-side javascript?

[Moonchild]

Well, there's javascript.options.main_thread_stack_quota_cap, but I don't really know the exact effects of that pref.
You could also try to force more gc on memory pressure with javascript.options.gc_on_memory_pressure, see if that helps any.

[Massacre]

I don't think it's possible to mitigate this on our end. If you run out of memory, then you run out of memory

Don't allow JS engine to allocate too much memory and force restart it (instead of entire browser) when it happens? Or it is not possible in current architecture at all?

It's much better to just stop all website JS and allow to shutdown gracefully if it could not be restarted than just crash with possible data loss (corrupted profile).

[Moonchild]

Don't allow JS engine to allocate too much memory and force restart it (instead of entire browser) when it happens?

it's not possible because the same JS engine drives the browser's application processing. It's not possible to perform a clean shutdown because it need a working JS engine.
I'm all for halting scripting on a page if it misbehaves, but that's not always possible, especially with how it's not in an interpreted state (the stack exhaustion occurs in one of the runtime components linked to the baseline compiler, as far as I know -- so to catch that you'd have to disable baseline compilation, and that would literally slow the engine down to unusable speeds).
I'm open to suggestions/patches to help mitigate this but it'll be difficult to do without killing browser usability.

[Kris_88]

I haven't fully figured out the problem, and I don't have much time, but if we remove the line "postMessage(nonce);", the browser works reliably and the check goes fine, and eventually the site page opens. I couldn't get the crash.

https://github.com/TecharoHQ/anubis/blo ... k.mjs#L120

And the same postMessage is in the second module, but this module seems not to be used on the Devuan site:
https://github.com/TecharoHQ/anubis/blo ... ow.mjs#L74

But during the check the progress indicator is not updated, since the deleted postMessage is used to periodically update it. I haven't figured out how Anubis receives messages from the worker, maybe the real cause of the problem is there.

Maybe this will help someone...

[Massacre]

it's not possible because the same JS engine drives the browser's application processing. It's not possible to perform a clean shutdown because it need a working JS engine.

Then, find a way to isolate browser core (trusted) from page-related (untrusted) scripts? Separate memory space (including stack), for example?

For now it looks like one huge security vulnerability - external scripts can crash the browser in a guaranteed way...

[Moonchild]

For now it looks like one huge security vulnerability - external scripts can crash the browser in a guaranteed way...

Malicious javascript can DoS browsers. it's a reality that has been true for decades.

I haven't fully figured out the problem, and I don't have much time, but if we remove the line "postMessage(nonce);", the browser works reliably and the check goes fine

Thanks, I looked into it and it does seem like they're doing something weird with their check there:

Code: Select all

        // send a progress update every 1024 iterations. since each thread checks
        // separate values, one simple way to do this is by bit masking the
        // nonce for multiples of 1024. unfortunately, if the number of threads
        // is not prime, only some of the threads will be sending the status
        // update and they will get behind the others. this is slightly more
        // complicated but ensures an even distribution between threads.
        if (
          (nonce > oldNonce) | 1023 && // we've wrapped past 1024
          (nonce >> 10) % threads === threadId // and it's our turn
        ) {
          postMessage(nonce);
        }

AFAICT, "nonce" is a counter that increases every loop by the number of threads (which they simply take navigator.hardwareconcurrency for -- which isn't very good because JS worker threads do not spread across all CPU threads "magically")
They set oldNonce to nonce, then increase the nonce.
if (nonce>oldNonce)|1023 is always going to be 1023, meaning it's always true.
(nonce >> 10) % threads is going to be the same number every block of 1024.
So, it looks like it's absolutely hammering the postmessage each iteration on the same worker...

[Kris_88]

So, it looks like it's absolutely hammering the postmessage each iteration on the same worker...

Indeed, it looks like they are overflowing the message queue.
And apparently this is indeed the cause of the problem.
Congratulations.

[Moonchild]

Looks like they'll work on fixing it. Thanks for pointing out the general area to look in and what ultimately caused the overflow symptom

[TheRealXeIaso]

Hi, main developer of Anubis here.

This is fixed in https://github.com/TecharoHQ/anubis/pull/880/. I will be tagging a release once this merges and my private test suite passes.

I think that this is the ultimate root cause of some of the weird and irreproducible issues that Pale Moon users have reported with Anubis. I was never able to reproduce them on my system because I use 64 bit hardware.

This fix is significant enough that I'm going to tag a release for this and post something on the blog about it. I would like to add automated tests for Pale Moon so that something like this never happens again. Is it possible to build a version of Pale Moon that is compatible with browser automation testing tools like Playwright or expose the Chrome Debugging Protocol so I can have it run at CI time?

Thank you all for your help. I'm sorry things have not been ideal.

[Massacre]

I think that this is the ultimate root cause of some of the weird and irreproducible issues that Pale Moon users have reported with Anubis. I was never able to reproduce them on my system because I use 64 bit hardware.

It does not depend on hardware. Actually, on Windows, 32-bit version is preferred to 64-bit because of being more compatible (plugins) and using less memory overall. I use some kind of Chromium for "web applications" anyway, and don't expect such abuse of system resources by external scripts.

[frostknight]

This is fixed in https://github.com/TecharoHQ/anubis/pull/880/. I will be tagging a release once this merges and my private test suite passes.

I think that this is the ultimate root cause of some of the weird and irreproducible issues that Pale Moon users have reported with Anubis. I was never able to reproduce them on my system because I use 64 bit hardware.

I have seen it happen on my end too and I use 64 bit. However the only one that does this to me currently is git.devuan.org

Not sure what is different about there implementation than say...

https://invidious.nerdvpn.de/

Its rather weird

[Moonchild]

Actually, on Windows, 32-bit version is preferred to 64-bit because of being more compatible (plugins) and using less memory overall.

64-bit is preferred to use with the "modern" web because of 32-bit limitations on overall memory, and there are some issues with the 32-bit target compiler that are a bit subtle but have been causing instability on old hardware. Unless you specifically have 32-bit only NPAPI plugins you need to use, I always do recommend using the version that matches your hardware.

I have seen it happen on my end too and I use 64 bit.

It's definitely possible but exceedingly less likely to happen. And I'm guessing Firefox has dodged this bullet because of the inherent slowness of IPC and asynchronous multi-process delays.

Not sure what is different about there implementation

They seem to set their config rather aggressively so it needs to perform more work for the challenge to pass, which makes it more likely that the message queue overflows with the bug that was there.

[Massacre]

64-bit is preferred to use with the "modern" web because of 32-bit limitations on overall memory

This is usually comes with very sluggish "modern" javascript that runs much better in some Chromium, so I prefer to not use "modern" web in Pale Moon (examples: youtube, "new" reddit interface, *.fandom.com)

and there are some issues with the 32-bit target compiler that are a bit subtle but have been causing instability on old hardware.

That's bad. Which specific hardware? Any workarounds?

Unless you specifically have 32-bit only NPAPI plugins you need to use, I always do recommend using the version that matches your hardware

Don't forget about forced AVX in official 64-bit builds.

[Moonchild]

That's bad. Which specific hardware? Any workarounds?

See reports of 32-bit crashes on this forum. I'm using a slightly older toolchain for 32-bit official builds to avoid this.

Don't forget about forced AVX in official 64-bit builds.

Insert dead horse beating remark here

[Moonchild]

Is it possible to build a version of Pale Moon that is compatible with browser automation testing tools like Playwright or expose the Chrome Debugging Protocol so I can have it run at CI time?

Unfortunately there isn't a ready way to run CI on Pale Moon. We explicitly don't support WebDriver in UXP, and our inherited Mozilla code for automated tests is not designed to test web compat (they are unit tests and reftests for browser internals). Taskcluster CI also depends on Mozilla infrastructure which we obviously don't have (nor have the resources for), so I'm afraid it'll be kludgy no matter what.

Your best bet is to set up an environment and launch Pale Moon by passing the URL to visit on the command-line, and observe.