A linux build framework for Pale Moon

Users and developers helping users with generic and technical Pale Moon issues on all operating systems.

Moderator: trava90

Forum rules
This board is for technical/general usage questions and troubleshooting for the Pale Moon browser only.
Technical issues and questions not related to the Pale Moon browser should be posted in other boards!
Please keep off-topic and general discussion out of this board, thank you!
Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-11-27, 06:40

Note that a new version of this framework has been created (click here). Please note that the original version (below) is now obsolete.

This all started when I first discovered Pale Moon for linux. I liked it immediately, but did not appreciate the hard-coded dependancy on dbus. So I set out to learn how to "roll my own". Soon I had a script for my ancient Atom netbook, and a script for my main desktop, and a script for my backup desktop. There was a lot of duplication, and I had problems tracking the differences, so I made a serious effort to separate the general infrastructure from the variable components. The result is a build environment that uses...
  1. infrastructure (necessary)
  2. a file called "mozconfig.txt" (necessary)
  3. a file called "customize" (optional)
This may be overkill for somebody who only wants to do a native build for their own machine, but it is nice for anybody who wants to do more. The infrastructure consists of a few directories and files...
  1. The top level (sub)directory is the main Pale Moon development directory. Let's call it "pmmain", but any legal name will do.
  2. Download the file "utils.tgz" attached to this post to "pmmain" (or whatever you called the main directory).
  3. Extract the file with the command...

    Code: Select all

    tar -xvzf utils.tgz
    This will create a subdirectory "utils"
  4. Clone the current branch from git. As of this post, the current branch is 27.0_RelBranch Note that this includes 27.0.x.y.z.etc. Stuff after the 2nd decimal point is not a seprate branch, although it may be released separately. There are 2 ways to do the cloning...
    • The hard way (Note: the following is actually one long line. It is not supposed to wrap)...

      Code: Select all

      git clone -b 27.0_RelBranch --depth 1 https://github.com/MoonchildProductions/Pale-Moon.git pmsrc
    • The easy way

      Code: Select all

      utils/getcode 27.0
    Either way results in a "pmsrc/" directory being created, with the source code. This can take a few minutes, depending on the machine, and the speed of your broadband connection.
  5. Create a subdirectory for each distinct version that you intend to build. For many people a "native" directory is sufficient. Any legal name will work, but it makes things easier if the directory name is descriptive, e.g. "native", "netbook", "machinexyz". For now we'll concentrate on the native build.

    Code: Select all

    mkdir native
  6. cd to the native directory, and symlink to the source directory

    Code: Select all

    cd native
    ln -s ../pmsrc pmsrc
  7. Symlink the files in the utils directory into the "native" directory...

    Code: Select all

    ../utils/makelinks
You have to construct a "mozconfig.txt" file. A "customize" file is optional. See the provided example files. Once you've done this, you can execute...

Code: Select all

time ./build
PLEASE READ THE readme.txt FILE FIRST!
Attachments
utils.tgz
Utilitiy files for Pale Moon build environment
(8.63 KiB) Downloaded 76 times
Last edited by trava90 on 2017-02-25, 07:07, edited 1 time in total.
Reason: Adding note about and link to "v2"
There's a right way
There's a wrong way
And then there's my way

half-moon

Re: A linux build framework for Pale Moon

Unread post by half-moon » 2016-11-27, 14:22

What's your issue with dbus?

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-11-27, 15:24

half-moon wrote:What's your issue with dbus?
I try to run a minimal system. The problem with bloat is that it sneaks up on you one piece at a time. The best analogy is "boiling a frog" by gradually raising water temperature. Another analogy is "creeping socialism", and there are others. One annoying example is Gnumeric. Over the years it has picked up HARD-CODED dependencies on goffice, ghostscript, harfbuzz, etc. I try to "draw a line in the sand" where possible, or at the very least minimize bloat. I remember running Gentoo linux on a Pentium 3 with 256 MEGAbytes of ram as late as 2007.
There's a right way
There's a wrong way
And then there's my way

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-07, 11:15

So what, exactly, does this allow me to do? I've just installed PM 25.8 via a SlackBuild script. I want to upgrade to 27. But I also share your aversion to bloat and coercive dependencies. I've been using Linux for a good while, but I'm just now getting serious about figuring out how to do more than click buttons on a package manager. I do install from source, but if it takes more than "./configure", "make", "make install" I typically move on to something else. Sorry if this is a rookie question. I am genuinely interested in what this is all about. For example, will it enable me to repackage PM for my chosen distro? What are the benefits/ramifications of avoiding dbus?

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-07, 15:42

Sasquatch wrote:I do install from source, but if it takes more than "./configure", "make", "make install" I typically move on to something else.
First, note viewtopic.php?f=37&t=14052 i.e. if your build runs into problems, please duplicate them in a standard binary. Otherwise, there may be reason to believe that the problem is specific to your build. Building Mozilla-based code is slightly different from building regular code. At its simplest, doing things manually involves 3 steps...
  • ./mach build (build the browser and associated libraries)
  • ./mach package (create a tarball containing the browser and associated libraries)
  • manually extract the tarball to "some/path"
From here you can launch it as "some/path/palemoon/palemoon". My build script, which launches "./mach build" and "./mach package", also runs various sanity checks, and allows for custom tweaks, such as using a non-default gcc version.
Sasquatch wrote:Sorry if this is a rookie question. I am genuinely interested in what this is all about. For example, will it enable me to repackage PM for my chosen distro?
You'll have to untar the tarball manually. Or you could set up a script to do it for you. One benefit is that if you build Pale Moon on your system, you know that it's linked against your version of glibc. Another benefit is that you can optimize the build for the specific machine it'll be running on. E.g. "-march=native" will take advantage of your cpu's full instruction set, and therefore run faster. I also do a personal build targetted at "-march=bonnell" for my ancient Atom netbook. The mainline linux build assumes a Pentium 4.
Sasquatch wrote:What are the benefits/ramifications of avoiding dbus?
One less library to load into memory. dbus is used for 3 purposes...
  1. To facilitate necko-wifi geo-location in Firefox (Not in Pale Moon)
  2. To enable WebRTC (e.g. Firefox's former "Hello" feature). There are some privacy concerns with WebRTC
  3. To enable "WakeLock". If you have a screensaver/screenlocker that talks to dbus, Wakelock is supposed to be able to prevent it kicking in while you're watching a long video.
There's a right way
There's a wrong way
And then there's my way

Zootal

Re: A linux build framework for Pale Moon

Unread post by Zootal » 2016-12-07, 15:55

I might be missing something...

time ./build
0:00.12 /usr/bin/gmake -f client.mk MOZ_PARALLEL_BUILD=4 -s
0:00.32 client.mk:201: /usr/local/src/palemoon/native/pmbuild/.mozconfig.mk: No such file or directory
0:00.51 Clobber not needed.
0:01.10 Adding client.mk options from /usr/local/src/palemoon/pmsrc/.mozconfig:
0:01.10 AUTOCLOBBER=1
0:01.10 MOZ_CO_PROJECT=browser
0:01.10 MOZ_MAKE_FLAGS=-j4
0:01.10 MOZ_OBJDIR=/usr/local/src/palemoon/native/pmbuild
0:01.10 OBJDIR=/usr/local/src/palemoon/native/pmbuild
0:01.10 FOUND_MOZCONFIG=/usr/local/src/palemoon/pmsrc/.mozconfig
0:01.10 /usr/local/src/palemoon/pmsrc/client.mk:304: *** Could not find autoconf 2.13. Stop.
0:01.10 gmake: *** [build] Error 2
0:01.12 0 compiler warnings present.

But it looks like we still need Mozilla's autoconf 2.13 dependency. A rather kludge of a fix is to symlink autoconf to autoconf213, since I have autoconf 2.69, but then I get:

time ./build
0:00.12 /usr/bin/gmake -f client.mk MOZ_PARALLEL_BUILD=4 -s
0:00.33 client.mk:201: /usr/local/src/palemoon/native/pmbuild/.mozconfig.mk: No such file or directory
0:00.51 Clobber not needed.
0:01.27 Adding client.mk options from /usr/local/src/palemoon/pmsrc/.mozconfig:
0:01.27 AUTOCLOBBER=1
0:01.27 MOZ_CO_PROJECT=browser
0:01.27 MOZ_MAKE_FLAGS=-j4
0:01.27 MOZ_OBJDIR=/usr/local/src/palemoon/native/pmbuild
0:01.27 OBJDIR=/usr/local/src/palemoon/native/pmbuild
0:01.27 FOUND_MOZCONFIG=/usr/local/src/palemoon/pmsrc/.mozconfig
0:01.39 Generating /usr/local/src/palemoon/pmsrc/configure using autoconf
0:01.76 build/autoconf/acwinpaths.m4:10: error: defn: undefined macro: AC_OUTPUT_FILES
0:01.76 build/autoconf/acwinpaths.m4:10: the top level
0:01.76 autom4te: /usr/bin/m4 failed with exit status: 1
0:01.76 gmake[1]: *** [/usr/local/src/palemoon/pmsrc/configure] Error 1
0:01.76 gmake: *** [build] Error 2
0:01.79 0 compiler warnings present.

Do I need to find/install the old version of autoconf, 2.13?

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-07, 20:50

Zootal wrote:Do I need to find/install the old version of autoconf, 2.13?
Yes. It is a hard dependancy. You must have autoconf-2.13 installed. It's part of the list at the beginning of the "readme.txt" file in the tarball I attached to the first message in this thread. That's why I asked, in bolded red text, that people read the "readme.txt" file. Linux distros should be able to handle multiple simultaneous versions of autoconf installed.

You also need >= yasm-1.01 if you want to install any or all of libjpeg-turbo, webM, and libvpx. You'll also need glib-dbus installed unless you disable dbus in the mozconfig. And you'll also need pulseaudio installed if you want to build in pulseaudio support. Note that you don't need pulseaudio to run Pale Moon with optional pulseaudio support, just for building with optional pulseadio support.

It is very strongly recommended that, where possible, you do NOT use "--with-system-libwhatever" options. That is begging for problems.
There's a right way
There's a wrong way
And then there's my way

Zootal

Re: A linux build framework for Pale Moon

Unread post by Zootal » 2016-12-07, 21:11

I read the readme, but I must have missed it because I was looking for something else at the time. Sorry!!!

I'll update (downgrade) autoconf and give it another shot....

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-07, 23:04

Zootal wrote:I'll update (downgrade) autoconf and give it another shot....
Most linux distros can have multiple versions of autoconf installed simultaneously. Check on the support forum for your linux distro. I primarily run Gentoo at home, and CentOS in a VM for building the SSE-only contributed version, and a personal version for my ancient Atom netbook. Both Gentoo and CentOS allow multiple versions of autoconf.
There's a right way
There's a wrong way
And then there's my way

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-07, 23:05

Walter Dnes wrote:Another benefit is that you can optimize the build for the specific machine it'll be running on. E.g. "-march=native" will take advantage of your cpu's full instruction set, and therefore run faster. I also do a personal build targetted at "-march=bonnell" for my ancient Atom netbook. The mainline linux build assumes a Pentium 4.
Well that right there is enough for me. I'm pretty sure a Pentium 4 is fairly different from the tiny AMD in this laptop. I'll have to tinker with it over the weekend. I'm starting to think the workstation I'm about to build will get an LFS install on it just so I can optimize every little detail.

And thanks for taking the time to answer my entry-level question.

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-07, 23:37

Sasquatch wrote:Well that right there is enough for me. I'm pretty sure a Pentium 4 is fairly different from the tiny AMD in this laptop. I'll have to tinker with it over the weekend. I'm starting to think the workstation I'm about to build will get an LFS install on it just so I can optimize every little detail.
You can get the benefits of cpu-optimization and dependancy control from Gentoo linux. With a few proprietary exceptions (e.g. Nvidia GPU firmware and Adobe Flash and Acrobat) Gentoo is usually built from source, but a lot of the grunt work is done for you automatically. Out of sheer curiousity, what is the output of...

Code: Select all

grep flags /proc/cpuinfo
...on your AMD? You should be able to run this from a terminal from any install or rescue CD or "Live CD"..
There's a right way
There's a wrong way
And then there's my way

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-08, 00:35

Walter Dnes wrote: You can get the benefits of cpu-optimization and dependancy control from Gentoo linux. With a few proprietary exceptions (e.g. Nvidia GPU firmware and Adobe Flash and Acrobat) Gentoo is usually built from source, but a lot of the grunt work is done for you automatically. Out of sheer curiousity, what is the output of...

Code: Select all

grep flags /proc/cpuinfo
...on your AMD? You should be able to run this from a terminal from any install or rescue CD or "Live CD"..
I tried Gentoo back when I first got into Linux. Gave up on it because it was beyond my skill set. Might give it another go. But I'm rather happy with my Slackware-based distro.

I've run cpuinfo before. I think it's a 300 series or something. I'll check again when I get home. Just got to work.

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-08, 15:36

Walter Dnes wrote:Out of sheer curiousity, what is the output of...

Code: Select all

grep flags /proc/cpuinfo
...on your AMD? You should be able to run this from a terminal from any install or rescue CD or "Live CD"..
Well that brought up a whole different set of information than I was expecting. Fascinating. So this is the stuff I would use to optimize installations on this laptop?

Code: Select all

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-08, 22:28

Sasquatch wrote:
Walter Dnes wrote:Out of sheer curiousity, what is the output of...
Well that brought up a whole different set of information than I was expecting. Fascinating. So this is the stuff I would use to optimize installations on this laptop?
This is a newer machine than you seemed to imply earlier. All the flags are implied by the exact cpu type. If you're doing the build on that machine then "-march=native" will be the way to go. It will imply all the corresponding flags. Since "sse4a" shows up, that means you have at least a "-march=barcelona" cpu., if not newer. And 2 "flags" lines means it's a dual-core.

Do you have gcc on that machine, or on a USB key or CDROM/DVD that can boot that machine? If so, what is the output of...

Code: Select all

gcc -c -Q -march=native --help=target | grep march=
...on the the AMD?
There's a right way
There's a wrong way
And then there's my way

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-08, 23:28

Walter Dnes wrote: Do you have gcc on that machine, or on a USB key or CDROM/DVD that can boot that machine? If so, what is the output of...

Code: Select all

gcc -c -Q -march=native --help=target | grep march=
...on the the AMD?

Code: Select all

-march=			btver1
So why would -march=native be the best? Doesn't that just tell gcc to figure out what the arch is? Why not specify the arch, I assume btver1? Or is "-march=native" sometimes used as shorthand like "foo"?

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-09, 02:07

Sasquatch wrote:So why would -march=native be the best? Doesn't that just tell gcc to figure out what the arch is? Why not specify the arch, I assume btver1? Or is "-march=native" sometimes used as shorthand like "foo"?
When you have more than 1 machine running building almost everything (like Gentoo) or even just Pale Moon, "-march=native" is portable across machines. It essentially says to optimize for this machine. I have 2 desktops and 2 linux note/net/books. Rather than trying to figure out what a machine is, I simply say "-march=native", and let gcc figure out what it's on. Computers were invented to do the grunt work. ;)

In order to run on as many machines as possible, Pale Moon has to build for a "lowest common denominator" cpu. This is a trade-off where you have to make the choice...
  • highly optimized binary that squeezes all the available performance out of your cpu, but doesn't run at all on older cpus
  • not-so-highly optimized binary that doesn't run as fast, but runs on more cpus
Note that saying "-march=btver1" implicitly includes all of...

Code: Select all

flags      : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat hw_pstate npt lbrv svm_lock nrip_save pausefilter vmmcall
And saying "-march=native" on your AMD machine is equivalant to saying "-march=btver1".

My ancient Atom netbook takes approximately 11 hours to build Pale Moon. The amazing thing is not that it takes so long, but rather that it can be done at all. It's a 30 or 40 minute build in a 32-bit VM on my desktop machine. When doing that build, I explicitly specify "-march=bonnell". gcc builds a bonnell-optimized binary. It won't even run in the VM, because the VM doesn't support the full Atom/Bonnell instruction set.

You can build Pale Moon optimized for your laptop one of 2 ways...
  • build it directly on the laptop with "-march=native" or "-march=btver1"
  • build it on your desktop with "-march=btver1". The resulting binary may not even run on your Intel desktop, but it will run on your laptop.
If the laptop is a bottom-end machine with only 2 gigabytes of ram and/or slow hard drive, remote compilation is the way to go. Gentoo linux has "distcc" and "distccd" which allows my netbook to offload compiling to a 32-bit VM on my desktop.
There's a right way
There's a wrong way
And then there's my way

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-09, 12:16

I've already noticed a huge difference in compile times, and abilities, between the laptop and the desktop. Which is pretty surprising considering the desktop is an obsolete HP with another dual-core AMD. A little better clock speed and double the RAM really make a difference. So I'll definitely be trying this on the HP. Do I just then replace the bin on the laptop with the new bin? Believe it or not, in over ten years of playing with Linux, including compiling from source, I have never upgraded a program from source. I've always been afraid of wrecking any configurations I had set. So all the programs I ever compiled myself just set there at whatever version they were when I installed them.

Thanks to this thread and your generous help I now feel emboldened. I already have a short list of programs I want to update and optimize. Looks to be a busy, and hopefully productive, weekend for me.

Sasquatch

Re: A linux build framework for Pale Moon

Unread post by Sasquatch » 2016-12-09, 15:22

I can't get the makelinks script to run. I cd'd into ./native and created the link to pmsrc. But when I tried "../utils/makelinks" I got an error saying no such file or directory. But if I cd into ./utils I can run the script with no problem. Except, obviously, it errors out rather than creating links in the same directory as the files it is trying to link. I end up with a link in ./native to ./native, but nothing else. I don't know what's going wrong. A missing dependency shouldn't cause problems with creating symlinks, should it? I doubt I'll be able to build on this laptop, but I at least wanted to see if I could create the environment. Bedtime anyhow. I'm probably too tired to be messing with this stuff at the moment.

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-09, 18:29

Sasquatch wrote:Do I just then replace the bin on the laptop with the new bin? Believe it or not, in over ten years of playing with Linux, including compiling from source, I have never upgraded a program from source. I've always been afraid of wrecking any configurations I had set. So all the programs I ever compiled myself just set there at whatever version they were when I installed them.
The combination of "./mach build" and "./mach package" results in the creation of an approximately 40 megabyte tarball in pmbuild/dist/ which you copy over to the target machine. You have to do a manual install as per the instructions in the first post of viewtopic.php?f=40&t=13530 (the Pale Moon SSE for linux thread).

The entire install is contained in the "palemoon" directory, wherever you install it. Doing an "rm -rf <dirname>/palemoon" removes the entire install, but mot the profile(s) in your home directory. For <dirname> substitute where you want to install. Popular destinations include /opt/ and /usr/local/ and $HOME/

Installing in /opt or /usr/local requires sudo or root permissions. The resulting install can run by any user on the system. Installing in $HOME does not require root/sudo permissions, but only that one user can run that copy of Pale Moon.
There's a right way
There's a wrong way
And then there's my way

Walter Dnes
Astronaut
Astronaut
Posts: 650
Joined: 2015-07-30, 20:29
Location: Vaughan, ON, Canada

Re: A linux build framework for Pale Moon

Unread post by Walter Dnes » 2016-12-09, 19:00

Sasquatch wrote:I can't get the makelinks script to run. I cd'd into ./native and created the link to pmsrc. But when I tried "../utils/makelinks" I got an error saying no such file or directory. But if I cd into ./utils I can run the script with no problem. Except, obviously, it errors out rather than creating links in the same directory as the files it is trying to link. I end up with a link in ./native to ./native, but nothing else. I don't know what's going wrong. A missing dependency shouldn't cause problems with creating symlinks, should it? I doubt I'll be able to build on this laptop, but I at least wanted to see if I could create the environment. Bedtime anyhow. I'm probably too tired to be messing with this stuff at the moment.
Here's what the directory layout should look like...

Code: Select all

<top level> pmmain/

Code: Select all

<2nd level> utils/   native/   machine2/   machine3/
 ("machine2", "machine3" etc are for any other builds you want to do)

From here, execute "utils/getcode 27.0". This creates pmsrc/ with latest 27.0.whatever source
Then "cd native"

Code: Select all

<3rd level is inside "native/">
Execute the 2 commands

ln -s ../pmsrc pmsrc
../utils/makelinks
That should get you going. Looking back, I think I can do a better setup, making things easier. I'll have to update the readme.txt to reflect the changes. Expect a version 2 soon. Sorry about not being clear enough. The hardest part of documenting stuff is saying the things "that go without saying".
There's a right way
There's a wrong way
And then there's my way

Locked