Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Users and developers helping users with generic and technical Pale Moon issues on all operating systems.

Moderator: trava90

Forum rules
This board is for technical/general usage questions and troubleshooting for the Pale Moon browser only.
Technical issues and questions not related to the Pale Moon browser should be posted in other boards!
Please keep off-topic and general discussion out of this board, thank you!
User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-13, 04:15

https://build.opensuse.org/package/show ... r/palemoon

It's not urgent right now, since the earlier successful Sid builds for 28.10.0 are still in its repository, but the issue will probably also migrate to Debian Testing, and might affect the next release. The build logs are available to anyone if you poke around the "failed" link, but do you want me to see if I can isolate the error messages?

bgstack was having the same problem on his Devuan unstable builds there the last time I looked...

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1498
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by athenian200 » 2020-07-13, 04:37

This is from your x86-64 build, but I notice something similar in the others too:

Code: Select all

[  168s] ../../coreconf/rules.mk:288: *** multiple target patterns.  Stop.
[  168s] make[8]: Leaving directory '/usr/src/packages/BUILD/platform/security/nss/lib/nss'
[  168s] make[7]: *** [../coreconf/rules.mk:67: private_export] Error 2
[  168s] make[7]: Leaving directory '/usr/src/packages/BUILD/platform/security/nss/lib'
[  168s] make[6]: *** [Makefile:451: private_export-nss/lib] Error 2
[  168s] make[6]: Leaving directory '/usr/src/packages/BUILD/obj-x86_64-pc-linux-gnu/config/external/nss'
[  168s] make[5]: *** [/usr/src/packages/BUILD/platform/config/recurse.mk:79: config/external/nss/export] Error 2
[  168s] make[5]: *** Waiting for unfinished jobs....
For whatever reason, it seems something is suddenly wrong with rules.mk having "multiple target patterns." I'm not sure what's different... a new compiler version, a new version of GNU Make, etc? Do you think it was a change in Debian's toolchain that caused this, or a change in UXP's code? I wouldn't expect the code has changed, so that would leave a toolchain issue. I wouldn't know where to start looking for that since I'm not on Linux myself these days, but it seems like figuring out what caused this to fail is important to fixing the problem.

From this, it looks like the builds started failing on June 30th?
Image
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Isengrim » 2020-07-13, 10:54

I don't see a specific error in the build log, just that it seems to fail somewhere in NSS.

This is just a shot in the dark, but I noticed that these warnings crop up right around where the build errors out:

Code: Select all

warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Moonchild » 2020-07-13, 11:42

Sounds like make has changed breaking parallel building (and I'm pretty sure our tree in some places relies on the jobserver being available, but don't pin me on that)
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Isengrim » 2020-07-13, 17:38

For reference, make 4.3 was released earlier this January. Announcement and relnotes here, including some details on breaking changes and deprecations. Debian Testing is still on make 4.2.1.

I tried spinning up a Debian VM, switching my package repos over to unstable (sid), and building Pale Moon. Interestingly enough I haven't run into any errors during configuration, building, or packaging. What version of make is installed on that machine, Steven?
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-14, 06:13

The OBS builds are done in the "cloud"--I'm not doing them locally. It will be whatever version of make is current in Sid. I checked, and Sid still has gcc-9 as the default compiler, so it's not gcc-10 causing the issue.

Maybe the jobserver messages are a red herring, since I think I see the same messages in the successful builds.

Re make: the log says:

Code: Select all

[   34s] [115/433] installing make-4.3-4
Hmmmm--Sid did an update of make on June 30, which is very suspicious, plus that jobserver change sure looks like it may have borked the builds now. 4.3 has been in Sid since late May, though:

Code: Select all

make-dfsg (4.3-4) unstable; urgency=high

  * Cherry picked [SV 58232] Disable inheritance of jobserver FDs for
    recursive make from ther development branch
  * Bug fix: "FTBFS: E: Build killed with signal TERM after 150 minutes of
    inactivity", thanks to Lucas Nussbaum (Closes: #963335).

 -- Manoj Srivastava <srivasta@debian.org>  Tue, 30 Jun 2020 00:58:36 -0700
I guess a workaround on the OBS would be a single threaded build. :(

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Moonchild » 2020-07-14, 09:31

stevepusser wrote:
2020-07-14, 06:13
I guess a workaround on the OBS would be a single threaded build. :(
And hope it doesn't take too long? Build systems tend to have timeouts to prevent "hung" builds.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

bgstack15
Fanatic
Fanatic
Posts: 121
Joined: 2018-01-22, 23:04

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by bgstack15 » 2020-07-14, 19:47

https://lwn.net/Articles/810384/ sounds relevant. For those who do not wish to navigate through the link, it is quoted here.
StefanBr on Jan 23, 2020 15:59 UTC wrote: > The jobserver protocol isn't hard to implement manually, even if not using make.

Unfortunately, the jobserver protocol is broken.

Job tokens are passed down via inherited file descriptors -

Caveat 1: If the called program is not $(MAKE) (verbatim), the file descriptors are closed on exec (FD_CLOEXEC), and the chain ends there. You can opt-in by adding a '+' in front of your command, but then you have to do that on every elegible command. Hint - none of the common Makefile generators does that.

Caveat 2: If you happen to call make indirectly (e.g. via a script called from the Makefile), unfortunately the '--jobserver-auth=<readfd>,<writefd>' will still be in the MAKEFLAGS env var although make had just closed the corresponding file descriptors. Two things can happen:

a) The fds no longer refer to valid file descriptors - make will detect this
b) The fds have been reused in the meantime - make will find valid file descriptors. If the fd is not readable (e.g. referring to a write end of a pipe), make will start busy looping. If the fd is readable, it will spawn a new job for every byte read (now figure what will happen if it points to a 2 kByte source file ...)
Also, other projects have run into similar-sounding errors. https://github.com/nodejs/node/issues/22457, where it was fixed by some reverting some Makefile changes in their code: https://github.com/nodejs/node/pull/22458/files. It is worth noting that nodejs had this issue before make 4.3 was released.

tl;dr: I'm no expert, but it looks like "$(MAKE)" might be needed instead of bare "make" invocations.

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-15, 03:34

I remembered that I can probably fix it by just adding the former working 4.3-3 (sources at snapshot.debian.org) version to my repo for Sid with an epoch of "1:" at the start of the version that will make it higher than 4.3-4.

Edit: well, that port of the older make also fails to build on Sid. :(

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Isengrim » 2020-07-15, 11:02

As mentioned previously, I'm able to build with make 4.3-4 on a vanilla sid VM without an issue.

What branch are you building? I have successfully built master, but will try building release instead to see if I can reproduce your issue.

Edit: Also what is the mozconfig you're using to build?

Edit 2: I checked my build of release on Sid and it still builds correctly, so now I'm wondering if it is something in our .mozconfigs that is different.
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-16, 04:27

Moonchild wrote:
2020-07-14, 09:31
stevepusser wrote:
2020-07-14, 06:13
I guess a workaround on the OBS would be a single threaded build. :(
And hope it doesn't take too long? Build systems tend to have timeouts to prevent "hung" builds.
Nah, I've had things like llvm and Chromium builds on the OBS take many hours, even with 8 threads, so PM is nothing compared to those. I just set up a single threaded build on Sid, and it seems to have got past the jobserver error.

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-16, 04:33

Isengrim wrote:
2020-07-15, 11:02
As mentioned previously, I'm able to build with make 4.3-4 on a vanilla sid VM without an issue.

What branch are you building? I have successfully built master, but will try building release instead to see if I can reproduce your issue.

Edit: Also what is the mozconfig you're using to build?

Edit 2: I checked my build of release on Sid and it still builds correctly, so now I'm wondering if it is something in our .mozconfigs that is different.
You can poke around the repo and download the PM debian tarball, which contains the mozconfig file:

https://download.opensuse.org/repositor ... _Unstable/

Single threaded PM builds seem to work fine in an experiment on the OBS--how many cores does your VM see? I tried to forwardport make 4.3-3 from snapshot.debian.org--that's the one that failed.

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Isengrim » 2020-07-16, 12:10

stevepusser wrote:
2020-07-16, 04:33
You can poke around the repo and download the PM debian tarball, which contains the mozconfig file:

https://download.opensuse.org/repositor ... _Unstable/

Single threaded PM builds seem to work fine in an experiment on the OBS--how many cores does your VM see? I tried to forwardport make 4.3-3 from snapshot.debian.org--that's the one that failed.
My VM is currently configured to use 2 processors and 8 GB RAM. In my mozconfig I have MOZ_MAKE_FLAGS="-j3"; I can try fiddling with this and other options to see if I can break my build. ;)
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Isengrim » 2020-07-16, 17:18

I tried building with the mozconfig you pointed to in the Debian tarball. (I also upped the number of processors for the VM to 4, just for fun.) The build still succeeded. I don't think the build system spits out a full log, so I tee'd it to a file - see attached.

Other than those jobserver warnings, I don't see anything suspect...
Attachments
log.txt
Debian Sid Build Log
(3.87 MiB) Downloaded 9 times
mozconfig.txt
Mozconfig
(1.71 KiB) Downloaded 12 times
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-17, 03:10

I can't really figure it out--sometimes the OBS has build problems that a VM on a local machine doesn't. I think they're using VMs on some kind of cluster of machines, but don't really know.

I can see it's not restricted to Pale Moon, either: https://build.opensuse.org/package/show ... kports/efl

Edit: and now it's working for EFL on Sid. PM builds are still failing.

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-18, 03:45

28.11.0 builds are now in my repo. I added a _constraints file to my palemoon-sid single threaded build which seems to have restricted the pool of builders for it to faster machines, so the builds were done in 2 to four hours--not completely unreasonable. Packaging on my laptop with sbuild took about 26 minutes, though a good part of that is having sbuild set up the build environment, including downloading and installing the many build-dependencies automatically, then cleaning up after itself after the build.

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-26, 03:41

I did come up with a little code for the debian/rules to use a detect a build on Sid and use a mozconfig that enforces a single build there, but will wait until the next PM update to avoid forcing another upgrade on users that won't make any difference to them. I suppose a fancy way to do that would be use sed to change just one mozconfig file on the fly...

Code: Select all

# Use a custom mozconfig for Sid to build with one thread.
# Builds on Debian derivatives may have to add their own release name.
distrelease := $(shell lsb_release -cs)
ifeq ($(distrelease),$(filter $(distrelease),sid unstable))
override_dh_auto_configure:
	cp debian/mozconfig-sid debian/mozconfig
endif

override_dh_auto_build:
	cp debian/mozconfig mozconfig
	make -f client.mk build

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by Moonchild » 2020-07-26, 14:24

Instead of copying you can simply "export MOZCONFIG=/path/to/mozconfig" and it will pick it up. i use that all the time for multiple builds from one tree, even doing them parallel.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
stevenpusser
Project Contributor
Project Contributor
Posts: 903
Joined: 2015-08-01, 18:33

Re: Hmmm--Pale Moon 28.10.0 is now failing to build on Debian Sid

Unread post by stevenpusser » 2020-07-27, 03:27

That's cool--so for a Sid build I just have it use the debian/mozconfig-sid file? That would be simpler.

Locked