New FreeBSD port

Talk about code development, features, specific bugs, enhancements, patches, and similar things.
Forum rules
Please keep everything here strictly on-topic.
This board is meant for Pale Moon source code development related subjects only like code snippets, patches, specific bugs, git, the repositories, etc.

This is not for tech support! Please do not post tech support questions in the "Development" board!
Please make sure not to use this board for support questions. Please post issues with specific websites, extensions, etc. in the relevant boards for those topics.

Please keep things on-topic as this forum will be used for reference for Pale Moon development. Expect topics that aren't relevant as such to be moved or deleted.
OlCe1

New FreeBSD port

Unread post by OlCe1 » 2020-11-13, 22:29

Hi,

I've just finished porting the Pale Moon/XUL code base to FreeBSD. In fact, I've been using Pale Moon for more than 6 months on this platform, without any problem.

But before packages are automatically built and distributed through official repositories, I would like to check with you that this is done in accordance with your licence and policy. I've indeed noticed a lot of activities around branding recently, including a change of the redistribution licence, so additional clarifications might be useful.

Main characteristics of this port
1. The build is done with unofficial branding.
2. The codebase is essentially FreeBSD-compatible, except for a few places, which have been minimally patched. The only notable exception is your particular version of jemalloc, which cannot work as is on recent FreeBSD because of an infinite recursion between "malloc" and "pthread_mutex_init". It may have worked before changes done for Issue #1307, but I didn't actually check beyond looking at commits related to it, in particular commit a954e19. I did not have time to fix this, so the port uses system's jemalloc instead.
3. Since Python 2.7 is deprecated and will be removed from FreeBSD ports, I had to patch the build system so that it builds with Tauthon. May be interesting to you as well. (I'm aware of viewtopic.php?f=5&t=24940)

The corresponding FreeBSD issue is 251117 (https://bugs.freebsd.org/bugzilla/show_ ... ?id=251117), where you can browse the patches.

I have the following questions:
1. Could you confirm that, given 1 above, there are no redistribution issues, or indicate what additional steps I should take? You'll probably want to look at the package description, which clearly states (I think) from the start that the build is not affiliated to Pale Moon or MoonchildProductions, although most of your code is included. www.palemoon.org is listed as the main reference site, which I would happy to change to some more relevant URL if you prefer (some specific forum or post explaining unofficial builds? repo.palemoon.org? nothing?).
2. Point 13 of the new redistribution licence seems to imply that using the unofficial branding provided is not eternally possible. I personally did not intend to have a new brand created, since the goal of this port is to stay very close to your code/building process (while allowing minor customization, though). Is this requirement really relevant in this case? If yes, I'll find someone that could do it (may take some time).
3. Patches mentioned in 2 (previous section) are all protected under #ifdefs, so should not disturb other platforms at all. Would you be willing to integrate these? If yes, I'll prepare an issue and pull request at repo.palemoon.org.

Thanks and regards.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: New FreeBSD port

Unread post by Moonchild » 2020-11-13, 22:49

Because of characteristic 1, there's no issue with licensing as you're not using official branding. Caveat emptor: as you noticed "browser" is not meant to become a perpetual thing (similar to the issue we had with New Moon basically being hijacked to mean one particular hackjob and not "all unofficial builds"). So no, I won't make an exception for FreeBSD to this intent, so you should make the effort to make your own branding if the build is going to be essentially different from what we are building on Windows, Linux and beta-Mac.

Not sure what issue you wanted to refer to but Pale Moon #1307 is definitely wrong as that's a PR and unrelated to jemalloc.

If you're otherwise opting to stay as close to official as possible, and willing to make proper upstream commits with proper #ifdefs and the likes, then I have no issue accepting these patches and even allowing official branding if the build configuration and setup is sane, i.e. not insisting on using system libs that are untested and unknown when there are known-good in-tree versions, or similar things that are disastrous to any mark of quality. Some of that might also hinge on how compatible Tauthon is with python 2.7 and what your build system patching entails. If it departs too much from the standard, then it's unlikely anything but your own branding will do.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2020-11-13, 23:15

I was referring to #1307 in UXP, not Pale Moon. Sorry for the wrong link.

As said, only system jemalloc would be used, the other libraries are your unpatched in-tree versions. Changes to build with Tauthon are minimal, and are done mostly to virtualenv so that modules and libraries are searched in the right paths. The other changes are necessary because of an implementation detail of Python 3.x/Tauthon (some "hidden" builtin needs to be present even in sandboxes). So basically, the build is as close as possible to the official one already, except maybe for some current default choices, like using GTK3 in preference to GTK2, and not building Sync support in. These are very easy to change anyway.

Didn't know that New Moon has been hijacked, although I was beginning to suspect it after reading other threads. Then this explains the recent change of policy.

I'll submit changes next week hopefully.

Thanks.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2020-11-13, 23:34

I personally want nothing to do with this effort. That being said, anything that relates to the Unified XUL Platform in regards of patches, pull requests, issues, etc will be held to the highest possible of standards and will be stringently examined and MUST be justified from every possible level forwards and backwards.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: New FreeBSD port

Unread post by Moonchild » 2020-11-14, 00:14

OlCe1 wrote:
2020-11-13, 23:15
system jemalloc
I'm sorry but that alone already is a sticking point. The memory allocator is at the heart of everything in the platform and our in-tree version (and as-configured in-tree!) is the only memory allocation library considered sufficiently stable for our torturous and atypical use. If that doesn't work for current versions of FreeBSD then it's a non-starter for official branding.
If using your own branding then of course you're free to keep your custom default choices, too.

I'm still fine with accepting platform patches if necessary for use on FreeBSD, but as pointed out by Tobin they have to be well-documented, rigorously checked and verified, and can in no way influence our primary (Windows/Linux) and secondary (Mac/SunOS) tier targets.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2020-11-30, 18:05

Hi,

Just a small update, since I've been a week late in reporting back with respect to my previous post.

I have not only polished patches against current head (some of the patches I used don't seem necessary any more, probably because I changed the compiler, not especially because of code changes), but I also spent a significant amount of time in modifying XUL's jemalloc in order to bootstrap it on FreeBSD. The goal here is to check if there are indeed advantages to it over system's jemalloc, and possibly to get a step closer to official branding. But testing is really the most important thing to me right now: While I trust that this allocator has been tuned for PaleMoon on Linux and Windows, to assume it will indeed work better on FreeBSD as well remains theoretical at this point, especially considering that it is based on an older version of jemalloc than FreeBSD's current one (thus increasing the a priori likelihood of differing behaviors).

With FreeBSD's system jemalloc, I have had an extremely stable experience so far. I've also noticed on very rare occasions big jumps in memory use as reported by the OS, but these generally went away quite quickly (probably when closing the offending tabs; don't remember the precise steps, it has been a while). I've had no crash at all. I'm generally a big user of tabs. So roughly comparing memory usage with XUL's allocator will be interesting. Fine-grained comparison, on the other hand, can be quite difficult, and I don't think I'll have the time to investigate it properly. Except if there are somewhere tests or benches that are designed to exercise these parts and are not too difficult to run.

As for PaleMoon/XUL specific allocator's requirements, I noticed there are tweaks to ensure that some higher-order bits are cleared in returned addresses on some platforms, apparently because the JS engine cannot deal with something else. Didn't investigate on whether this constraint depends on something else (arch, OS, configuration, etc.), or is just always present. In any case, it would not be difficult to code a simple wrapper around the system allocator ensuring this property is always obeyed.

Unfortunately, my processor broke just as I finished the work, so I did not have the opportunity to test and submit as I had planned. For technical reasons, I won't be able to access the data quickly, and I'll have to wait a week or two before I can report back again with the proposed code.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2020-11-30, 18:11

Well you can show us a repository but do not open a pull request until you are instructed to.

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2020-12-14, 10:18

Hi,

Please have a look at OlCe1/UXP, FreeBSD-support branch.

Happy to answer questions.

Regards.

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2021-01-15, 14:17

Hi,

So now that FreeBSD support went in, I would like to see with you if I can enable official branding, and how. So are there precise requirements, beyond "sane build environment"?

As for the current situation, there is one thing you may not like: I build Pale Moon with GCC 9, but in the end C++ code is linked to FreeBSD's libc++. Nonetheless, this is what I have tested, and seems to be perfectly stable.

I don't think it will be possible to link Pale Moon to libstdc++ on FreeBSD reasonably simply. The reason is that, in standard FreeBSD setups, e.g., when using pre-built packages, most C++ code is built using Clang/LLVM, and C++ code is linked against the OS' libc++. And in particular here, this is the case for "graphite2", a requirement of "harfbuzz", and "libgraphite2.so" ends up being mapped into Pale Moon's process at run-time. When Pale Moon's code proper is linked against libstdc++, I get segmentation faults at startup (which was not entirely obvious, since both libc++ and libstdc++ can coexist in a process under certain conditions; but here they are apparently not met; more importantly, I think that doing this is asking for trouble, now or later).

So either you are OK with the build with GCC 9 and link with FreeBSD's libc++ as it is (again, I've not had a single crash for months with that setup), or you reject it, and then I will need to explore building Pale Moon with clang(++) instead. In which case, I would like to know first if this would be acceptable for official branding? Some comment in the merged PR hinted that some Linux distributions were doing that, but is this officially approved? Very early, I built Pale Moon using system's Clang/LLVM 8 but as far as I remember, I got crashes with the resulting libs and executable. Probably some specific version will be required for the build. I just hope that there is indeed one that can work if I have to go down this route.

Regards.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2021-01-15, 15:02

Let us see your mozconfig.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35475
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: New FreeBSD port

Unread post by Moonchild » 2021-01-15, 16:15

OlCe1 wrote:
2021-01-15, 14:17
I don't think it will be possible to link Pale Moon to libstdc++ on FreeBSD reasonably simply.
OlCe1 wrote:
2021-01-15, 14:17
So either you are OK with the build with GCC 9 and link with FreeBSD's libc++ as it is
I think this situation classifies as required changes to come to a positive build on the target operating system, which means it's not a hindrance for official branding.

But otherwise, yes:
New Tobin Paradigm wrote:
2021-01-15, 15:02
Let us see your mozconfig.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2021-01-15, 17:29

Ok, here it is:

Code: Select all

_BUILD_64=1
_GTK_VERSION=3
mk_add_options MOZ_OBJDIR=/usr/ports_12_2/www/palemoon/work/pale-moon/pmbuild

ac_add_options --enable-application=palemoon
ac_add_options --enable-optimize="-w -O2"
ac_add_options --enable-default-toolkit=cairo-gtk$_GTK_VERSION
ac_add_options --enable-jemalloc
ac_add_options --enable-strip
ac_add_options --enable-devtools
ac_add_options --disable-eme
ac_add_options --disable-webrtc
ac_add_options --disable-gamepad
ac_add_options --disable-tests
ac_add_options --disable-debug
ac_add_options --disable-necko-wifi
ac_add_options --disable-updater
ac_add_options --with-pthreads
ac_add_options --enable-phoenix-extensions

ac_add_options --enable-official-branding
export MOZILLA_OFFICIAL=1

export MOZ_PKG_SPECIAL=gtk$_GTK_VERSION

# Don't bother compressing, mach's package is just intermediate file
export MOZ_PKG_FORMAT=TAR


# Below this line, build options depending on port options, as well as make and
# build flags, will be appended.
ac_add_options --enable-alsa
ac_add_options --disable-pulseaudio
The end of the file is where extra lines are appended depending on chosen options, for non-default options. If possible, I would like to enable ALSA instead of Pulseaudio as the default backend.

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2021-01-15, 17:43

No, wait, some stuff is missing to force linking against libc++.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2021-01-15, 18:43

You obviously don't understand that we are bypassing some of the mozilla packaging so your export of MOZ_PKG_FORMAT won't mean shit. Given you are creating a system package just issue mach stage which will simply stage the shit in dist/palemoon and grab it out of there so you don't have to have some intermediate unpackage or overriding shit.

Also, since I assume FreeBSD is predominantly or exclusively GTK3 so just set it directly to --enable-default-toolkit=cairo-gtk3 and don't export MOZ_PKG_SPECIAL at all.. It makes no sense.

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2021-01-15, 23:04

You obviously don't understand that we are bypassing some of the mozilla packaging so your export of MOZ_PKG_FORMAT won't mean shit.
Don't you know that MOZ_PKG_FORMAT does change the format of the package produced by "mach package"? And "mach package" is the route I chose even before the Linux build instructions were updated to say to run it (they were saying to copy some directory). I don't want to second guess the changes that will inevitably be made in the future, and there was no official instruction on what should be copied and installed, except now there is "mach package".
Given you are creating a system package just issue mach stage which will simply stage the shit in dist/palemoon and grab it out of there so you don't have to have some intermediate unpackage or overriding shit.
If you're saying that "mach stage" does exactly the same, minus the tarball creation, then maybe... but at the very least it would have to be documented accordingly on the build instructions page, so this is not changed lightly. Considering the time it takes to build the browser, the packaging overhead of "mach package" is anyway irrelevant. And I'm not going to exchange some irrelevant improvement for extra maintenance burden.
Also, since I assume FreeBSD is predominantly or exclusively GTK3 so just set it directly to --enable-default-toolkit=cairo-gtk3 and don't export MOZ_PKG_SPECIAL at all.. It makes no sense.
It's the trend, but it is not yet. I'll do that, this will simplify things a bit.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2021-01-15, 23:09

Right, FreeBSD. Which is ignored and falls back to original routines. Everything I said is valid aside from that. But yeah, use mach stage and save some hassle.

WELL for future reference this is what we are doing on Windows Linux and SunOS:
http://xref.palemoon.org/moonchild-central/source/palemoon/build.mk#8
http://xref.palemoon.org/moonchild-central/source/palemoon/installer/Makefile.in#117
http://xref.palemoon.org/moonchild-central/source/platform/toolkit/mozapps/installer/packager-uxp.mk

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2021-01-16, 00:18

Additional variables in ".mozconfig":

Code: Select all

export CPP="cpp9"
export CC="gcc9"
export CXX="g++9"
export CFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing"
export CXXFLAGS="-O2 -pipe -fstack-protector-strong -nostdinc++ -I/usr/include/c++/v1 -nodefaultlibs -lc++ -lcxxrt -lm -lc -lgcc -pthread -lpthread"
export LDFLAGS="-lgcc -fstack-protector-strong -L/usr/local/lib/gcc9"
export LD="/usr/local/bin/ld"
export AS="/usr/local/bin/as"
export AR="/usr/local/bin/ar"
export RANLIB="/usr/local/bin/ranlib"
export OBJDUMP="/usr/local/bin/objdump"
export NM="/usr/local/bin/nm"
Now that I'm compiling against a more recent libc++, I'm getting an error in "gfx/graphite2/src/CmapCache.cpp" in <cstdlib>, because mozilla/mozalloc.h is included in the chain, and finishes including <exception>, which needs "abort", defined later in <cstdlib>... So I guess more work is needed.

I'll also explore compiling with clang 10.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2021-01-16, 00:34

How about you don't and perhaps leave off that garblemesh while you are at it.

OlCe1

Re: New FreeBSD port

Unread post by OlCe1 » 2021-01-16, 00:58

Solution for last error is here: Mozilla's bug #1594027. Build is progressing.

No more time right now, I'll test the result next week.

New Tobin Paradigm

Re: New FreeBSD port

Unread post by New Tobin Paradigm » 2021-01-16, 01:09

I thought it was already finished. So your pr waa incomplete?

Locked