Pale Moon User Interface Freezes

[linuxrocks123]

Platform: Linux x64 GTK2

After the browser has been open for a day or so, the UI will regularly freeze for several seconds at a time. During these freezes, one CPU core will be spiked at 100%, and keystrokes and clicks will be buffered and will be processed once the freeze has completed. Once these freezes begin, they happen so frequently that the browser must be restarted to restore an acceptable user experience. This failure mode seems to happen more frequently on my AMD-based laptop than on my Intel-based desktop machines.

My guess is that it's the JavaScript garbage collector or something similar. Building Pale Moon seems like an involved process, but, if you can point me to a Linux binary that has debugging symbols included, I'll run it through GDB and give you a backtrace of the program state when the freezes happen.

[biopsin]

Yea I notice it only and mainly after visiting github; cpu bounces from 10% to 50% with all tabs closed until browser restart.

[scaffold]

I see it too. After some time, after visiting some sites, PM is freezing constantly. I'm pretty sure it is related to memory. GC or something. I see it at gkrellm showing memory utilization. There is always kind of memory freeing when freezing ends.

I'm using linux too. And have enough memory. I have impression there is some internal memory limit in PM. It is hit and freezing begins. Maybe there are some about:config variables I can change to give PM more memory?

[Enobarbous]

I see it too. After some time, after visiting some sites, PM is freezing constantly. I'm pretty sure it is related to memory. GC or something ... Maybe there are some about:config variables I can change to give PM more memory?

Oh, this is an old problem that occurs periodically and has no reliable solution (well, except for not using certain sites, yes:))
As a partial solution - have you tried settings from here and/or using avx build?
From personal experience, I can't say that it solved the problem at all, but freezes became less...

[suzyne]

have you tried settings from here and/or using avx build?

I used to get the freezing behaviour and what worked for me is never visiting https://outlook.com/ in Pale Moon, using the AVX2 version and having these memory settings. I no longer get any UI freezing that I would experience when running all day and the evening (I restart each morning so my situation is not identical) but for me the problem is resolved.

[Moonchild]

Disabling incremental GC is not recommended. That would actually increase the chance of long pauses/freezes as GC is running.
There was previously an issue with the compacting that has since been resolved. Generational GC should be stable as well and I haven't seen this leading to UI freezes myself even on aging hardware, but YMMV depending on the specifics of the scripting in use. Not using generational GC decreases efficacy of GC especially for small and short-lived allocations so may lead to stuttering in games or high-refresh scenarios.

[suzyne]

Disabling incremental GC is nor recommended...

I disabled those in the past, when I read somewhere on the forum that it is worth experimenting with those options when there are problems with UI freezing etc.

I now remember that there was an update which addressed the memory issues, but I forgot to go and change them back to the defaults.

All I know is that Pale Moon is currently working really well for me (provided I stay clear of Outlook!) with my current settings and I don't experience the problems that I had before.

But I will set the memory GC options back to the defaults, since that is recommended.

[P_A_Semi]

The "freezing" (I experienced often) is caused by vain repeated GC, when PaleMoon wastes most of its memory address space, soon before it outright crashes... (I do have tools to observe stack and what it does...)
It is especially annoying, that it sits minimized during a night doing nothing, but just when I open the icon and need to do something, it starts to repeatedly "trace edges" for GC and is almost frozen...

It can be solved by disabling javascript.options.gc_on_memory_pressure , but then it may crash a little sooner...
With or without this option it soon crashes anyway - I usually run Palemoon at most 3 or 4 days until it crashes, but last week it crashed daily, every single day, regardless of this option... I visit just 12, sometimes 20 web-pages, mostly simple WordPress sites, not more, as counted by "thumbnails" folder, which I empty after the crash to see next time, how many pages it visited before the crash... But I reload them multiple times by Alt+D, Enter to see comment replies, not using F5 to stress web-servers with vain refresh attempts, and emptying Browser Console often... The memory is not properly released when the page is left or reloaded...

Maybe the wasting of address space could be little solved by enabling memory.free_dirty_pages ? But it doesn't seem very different...

The problem is caused by wrong strategy of Memory Manager... It was already in old Firefox (45), just Palemoon crashes sooner, wasting more address space for same working set footprint... (on Windows 7 32-bit if it makes any difference?)

The wrong strategy is to keep 1M address space regions, until all 4k pages are released. They are never released, there is a lot of memory leaks, holding 4k pages, holding 1M address space regions, until there is nothing left... The correct strategy would be using memory with 4k granularity for small blocks and reusing them, not expecting the 1M regions to ever be completely free... Thereby it must not isolate different "tabs" each in its own set of 1M regions, it must be isolated at most on 4k pages...
(Was it designed by some optimistic fresh graduate, who has been taught: memory leaks must not happen! Then not expecting them to happen, then not having a strategy to handle the leaks gracefully...)
(If you are programmer, see attachments on this post, a very typical situation, address space full of pages like a cribble with occasional memory leaks holding 4k pages, then not releasing and not reusing the whole 1M memory regions, until the address space is full... The E5 byte is filled into freed memory blocks... The screens were taken, while it was waiting in app-crash dialog for wasting all 2Gb of address space, using 785Mb working set - why does it have 4k holes all over the address space without reusing them?! Firefox 45 usually crashed for same reason, when it had over 900Mb working set, maybe it had more leaks... Please remember: Leaks happen! Memory manager must handle that better...
The third attachment is an example stack-view of GC "freezing" or slowing down Palemoon in vain...)

(Just a side note - I have a very thorough experience writing various memory managers (in Pascal+Assembler, not in C) and I do know what I write about...)

πα½

[athenian200]

The problem is caused by wrong strategy of Memory Manager... It was already in old Firefox (45), just Palemoon crashes sooner, wasting more address space for same working set footprint... (on Windows 7 32-bit if it makes any difference?)

The wrong strategy is to keep 1M address space regions, until all 4k pages are released. They are never released, there is a lot of memory leaks, holding 4k pages, holding 1M address space regions, until there is nothing left... The correct strategy would be using memory with 4k granularity for small blocks and reusing them, not expecting the 1M regions to ever be completely free... Thereby it must not isolate different "tabs" each in its own set of 1M regions, it must be isolated at most on 4k pages...
(Was it designed by some optimistic fresh graduate, who has been taught: memory leaks must not happen! Then not expecting them to happen, then not having a strategy to handle the leaks gracefully...)
(If you are programmer, see attachments on this post, a very typical situation, address space full of pages like a cribble with occasional memory leaks holding 4k pages, then not releasing and not reusing the whole 1M memory regions, until the address space is full... The E5 byte is filled into freed memory blocks... The screens were taken, while it was waiting in app-crash dialog for wasting all 2Gb of address space, using 785Mb working set - why does it have 4k holes all over the address space without reusing them?! Firefox 45 usually crashed for same reason, when it had over 900Mb working set, maybe it had more leaks... Please remember: Leaks happen! Memory manager must handle that better...
The third attachment is an example stack-view of GC "freezing" or slowing down Palemoon in vain...)

(Just a side note - I have a very thorough experience writing various memory managers (in Pascal+Assembler, not in C) and I do know what I write about...)

πα½

This is a very interesting comment. We could probably use some help from someone who knows more about memory management and garbage collection, honestly.

So, is the problem more than likely in our jemalloc implementation, or is this more of a GC issue somewhere in the JS engine? I definitely get the impression there are a lot of leaks that just pile up over time, but I don't have a clear idea how we would stop them from happening or deal with them better than we currently do. The only real experience I have touching that code is from when I tried porting this codebase to Solaris/illumos.

I also have a little bit of experience with hex editors, though I never used them to view memory before. I have seen pointer tables in a hex editor before, though, and this reminds me a bit of that. From what I can tell though, it definitely looks like the free space isn't being organized or used as well as it could be, but I'm out of my depth for sure.

As a user, there are definitely some quirks that I've just gotten used to that remind me of how older versions of Firefox tend to just balloon in memory usage over time and have to be restarted regularly, but I never really thought about why and just assumed it was inevitable with this codebase unless we can find an expert to fix this stuff.

[KlarkKentThe3rd]

Finally, a thread about this! Maybe something good will come out of this conversation.

[ChrisCat]

I notice this happen more readily if I leave open the chat windows in youtube or twitch livestreams (including chat replays). I'll start getting periodic freezes that get longer over time. It's happens so regularly for me that I've grown a habit of closing stream chats as fast as I can when going to such a video, which mitigates the problem. Closing the windows seems to stop it from getting worse, but it'll continue to regularly freeze once it starts.

Relatedly, in about:memory and clicking Measure under "Show memory reports", it'll show results for youtube and twitch long after those windows have been closed. None of the buttons under "Free memory" to minimize memory or do a GC sweep have an effect on getting rid of them. The only way to get rid of those entries and stop the freezing is to restart the browser.

[Moonchild]

If it was what you think it is which was addressed in Firefox 45, then we already have that code since our fork point for the current iteration of the platform code was later than that... So that can't be it.

[P_A_Semi]

So, is the problem more than likely in our jemalloc implementation, or is this more of a GC issue somewhere in the JS engine? I definitely get the impression there are a lot of leaks that just pile up over time, but I don't have a clear idea how we would stop them from happening or deal with them better than we currently do.

After inspecting some source code from developer.palemoon.org for version 29.4.6 - I will not correct it, because I don't have any of the Prerequisities mentioned there beside Tea & Time...
(Do you still have a programmer maintaining that part of code? Last time I've been writing code in C was in mid-90's, then using Pascal+Assembler, or Scripts...)

There are two problems...

The memory is being eaten most probably by jemalloc... (I still need to verify that...)

Then, when there is a "pressure", the program "freezes" (is very slow), and it has on stack GCForReason() call with reason MEM_PRESSURE.
This is called from /platform/dom/base/nsJSEnvironment.cpp from nsJSEnvironmentObserver::Observe twice with JS::gcreason::MEM_PRESSURE, which should be guarded with some time-stamp, when it was last invoked, because "low-memory-ongoing" probably does not work well... (maybe better would be to guard against this state inside /platform/js/src/jsgc.cpp JS::GCForReason() or in GCRuntime::gc()... with reason==MEM_PRESSURE check timestamp of last invocation and refrain from doing that sooner than one or few seconds from last call, or if the previous gc did not free much, set some flag to prevent MEM_PRESSURE and clear that flag in timer... because if the pressure gc occured and did not free much, it is highly likely it will be invoked very soon again and again, because someone else has eaten the memory...)
(possibly also in gc/Allocator.cpp GCRuntime::gcIfNeededPerAllocation should also check some flag zone()->gcUnsuccess after checking usage.gcBytes() > ... , set if last such gc did not free much memory and the condition remained after last gc, cleared in few seconds timer, so it does not do it on each allocation in case of "pressure"...?? I am not sure if this actually happens, if I cannot place a breakpoint there...)

-----
The pages that overrun have a list at 00400B00 (one of first allocations after program start) pointed from 00400AC4, there is a long array of page pointers...
These pages are 1M in size and usually (not always) start with 40 00 40 00 , B4 00 40 00 , B4 00 40 00 ... and they don't have any structure at end of the 1M page, which GC chunks should have...
Free space in these pages is filled with E5 byte as in jemalloc() ...

-----
For jemalloc, before studying that, I would suggest:
- Prefer reusing existing released memory pieces and blocks over getting new ones
- Have a list of partially free 1M memory regions (chunks) where pages (arenas) were decommited... (either doubly-linked list in those chunks, or sequential array elsewhere. Entries in that list have at least 4k decommited memory for an arena)
- Before acquiring new chunk for a new arena, pick some from the list of partially free 1M chunks
- Where the chunk is released by VirtualFree/MEM_RELEASE, you must also remove it from that new list of partially free pages for reuse (which is simpler for doubly-linked list (next,prev) as you don't need to use indexOf to find it in the array list... needs propper locking for thread concurency...)

Consider, that when you use VirtualFree with MEM_DECOMMIT, you release physical memory, but not the Address space, which then congests, when it reaches 2G. (i.e. you still should decommit/unmap memory, but that is not enough, you should reuse them before acquiring new ones)

The existing partially free Arenas should be better reused, when acquiring new memory piece...
In arena_dalloc_small(), maybe arena_run_tree_insert() does not work well?? Or run->nfree==1 does not occur at this point?? Or I don't know, why it does not work as expected?

πα½

[P_A_Semi]

If it was what you think it is which was addressed in Firefox 45, then we already have that code since our fork point for the current iteration of the platform code was later than that... So that can't be it.

Firefox 45 which I still use beside Palemoon has the same problem, just it occurs a little while later, not much later with same browsing...
There the same 1M-page list is at 00500BC0 pointed from 00500B98 ... (in Bootstrap block), and the free pieces are filled there with E5 byte... These pages usually start there with bytes 40 00 50 00 , C4 00 50 00 , C5 00 50 00 , ...

(I have over 400 tabs opened in that Firefox 45 in two windows, most of them inactive after last crash and session restore, and I use it for administering various web-pages, the sql editors have sql in the window.history, or for my own web-pages which are well compatible with even older Firefox versions than this one... and probably few hundred tabs I already don't use but didn't close them...)

In my behavior I described above is missing, that in Palemoon I reload WordPress page by Alt+D,Enter, then save it into MHT, then I post a comment, then save it into MHT... Usually less then ten such tabs, reloaded alternately... It fills 2G address space in 1 to 3 days, about 50 mht files (and page reloads) per day...

πα½

[Kris_88]

If we have ghost windows, then this is not a low-level memory allocation/deallocation problem. This is a problem with freeing java objects. This is a completely different level.
Although there are probably leaks at low level too...

[P_A_Semi]

... This is a problem with freeing java objects. ...

That, what happens in my case (both Palemoon and Firefox45) is, that low level wastes a lot of address space (and memory), and then GC from Javascript almost freezes program, because it tries to release some JS memory because of "pressure", but the pressure is caused elsewhere, it releases almost nothing, but tries soon again and again in vain...

(I just verified in my case, that the excessive 1M blocks of memory are not related to that, what jsgc.cpp manages... disassembled mozjs.dll, compared with cpp sources, compared with runtime memory... GC memory blocks have some struct at offset FFFA0 bitmap, at FFFC0 chunk->info.next, etc... this is not the case with the excessive 1M blocks, filled with E5 byte around "leaks"...)

πα½

[Kris_88]

Then it’s logical to look for leaks. Maybe add some kind of logging? If we have a table "malloc call address" - "address of allocated block", then it will be easier to find the cause of the leak. I haven't messed around with this part of the code. Maybe something similar already exists and just needs to be enabled?

[Kris_88]

Or we can try modifying malloc so that it allocates an extra 8 bytes and stores its call address there. Then inside the block we will see from which part of the code it was allocated.

[Kris_88]

How to get a return address.

https://learn.microsoft.com/en-us/cpp/i ... dfrom=MSDN
http://gcc.gnu.org/onlinedocs/gcc/Return-Address.html

[athenian200]

If this turns out to be a jemalloc problem, the easiest way to solve it might be trying a different malloc instead, or even trying to update the jemalloc code. I don't think Mozilla even wrote that code initially, and some other applications do experiment with trying a different malloc. I think changing the way jemalloc works would be as likely to introduce issues as just updating it or using a different malloc, since any quirks the GC depends on would be changed if we start messing with the internals of jemalloc.

It's worth noting that we officially recommend building with jemalloc mostly because our JS engine and its GC is designed around its quirks... but I have heard reports that on operating systems other than Windows and Linux, sometimes the system allocator actually works better than jemalloc, though it really, really varies and I haven't verified this for myself. I know for a fact that glibc's malloc sucks, as does that of Windows 7 and earlier, which I think is why jemalloc became a thing in the first place. While Windows 10 and later do have a much better malloc implementation from what I hear, we can't rely on that because a lot of our users prefer older Windows.

The fact is, though, jemalloc is basically just some old FreeBSD libc code that got used as a cross-platform malloc by Mozilla at some point, and we inherited that decision without ever really examining it ourselves... mostly because I don't believe we actually have the expertise to review a decision like that, not being experts on low-level memory management.

Given that context, I am not sure about whether it makes sense to dig down into these kind of low-level problems and try to fix them ourselves if they're actually jemalloc problems, or whether we should try replacing or updating the malloc library first and see if a lot of the problems go away.

If we want to update jemalloc, there's a repo here that we can try pulling in changes from, any bugs in our implementation may have been fixed already.

https://github.com/jemalloc/jemalloc

Or, if we want to try a different malloc, I've heard good things about mimalloc, which is written by Microsoft, but is cross-platform and MIT-licensed:

https://github.com/microsoft/mimalloc

And with Solaris specifically, I know a few people who swear libumem is a really good allocator, and there are ports of it to other platforms, though those may not be well-tested enough for our use case. There's also tcmalloc, which is developed by Google, and basically there are a lot of choices to experiment with. I assumed our in-tree jemalloc was stable and worked really well for us, but kind of seems like it could be one of those things that just works well enough we haven't bothered to change it, and perhaps it only works better relative to our (probably somewhat bitrotted) code paths for using system allocators, but isn't actually the absolute best choice...

There could be some technical reason why only this version of jemalloc would work for us, but if there is, I'm not aware of it... it seems like the main downside to using a newer/different malloc is we might have to adjust some assumptions made by the GC to reflect the fact that we're using something else, because currently I think it just assumes we use our in-tree jemalloc, complete with known bugs/issues, and might bork on the use of anything else as a result.