Usefulness of development files in package

Users and developers helping users with generic and technical Pale Moon issues on all operating systems.

Moderator: trava90

Forum rules
This board is for technical/general usage questions and troubleshooting for the Pale Moon browser only.
Technical issues and questions not related to the Pale Moon browser should be posted in other boards!
Please keep off-topic and general discussion out of this board, thank you!
User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 08:47

In our Pale Moon packaging for Nixpkgs we noticed that mach install places some development-related files under the configured prefix.
  • include
  • share/idl
  • lib/palemoon-devel-$VERSION
I know they're not completely comparable anymore but these files are called "SDK cruft" in our Firefox packaging and get subsequently removed. Due to the size impact and unknown usefulness of these files to end-users/developers we PM maintainers decided to delete them for now as well. This brings the package size and contents much closer to the pre-compiled Linux tarballs you distribute. We'd like to know if this was a bad decision and these files do have a use to e.g. extension developers though, in which case we'd prefer to reinstate and move them to a separate-but-related dev package instead.

Link to relevant discussion (GitHub).

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Usefulness of development files in package

Unread post by Moonchild » 2021-05-03, 08:56

How exactly are you building? what is your mozconfig?
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Re: Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 09:07

How exactly are you building?

Code: Select all

# Configuring
export MOZCONFIG=$PWD/mozconfig
export MOZ_NOSPAM=1

echo > $MOZCONFIG '(mozconfig contents here)'

# Building
./mach build

# Installing
./mach install

substituteInPlace ./palemoon/branding/official/palemoon.desktop \
  --replace 'StartupWMClass="pale moon"' 'StartupWMClass=Pale moon'
desktop-file-install --dir=$out/share/applications \
  ./palemoon/branding/official/palemoon.desktop

# Install official branding icons
for iconname in default{16,22,24,32,48,256} mozicon128; do
  n=''${iconname//[^0-9]/}
  size=$n"x"$n
  install -Dm644 ./palemoon/branding/official/$iconname.png $out/share/icons/hicolor/$size/apps/palemoon.png
done

# Remove unneeded SDK data from installation
# TODO: move to a separate output?
rm -rf $out/{include,share/idl,lib/palemoon-devel-${version}}
EDIT: Plus some automatic wrapping & symlinking post-installation so PM works as expected.
what is your mozconfig?

Code: Select all

# Clear this if not a 64bit build
_BUILD_64=${lib.optionalString stdenv.hostPlatform.is64bit "1"}

# Set GTK Version to 2 or 3
_GTK_VERSION=${gtkVersion} # Input to build process, 3 by default

# Standard build options for Pale Moon
ac_add_options --enable-application=palemoon
ac_add_options --enable-optimize="-O2 -w"
ac_add_options --enable-default-toolkit=cairo-gtk$_GTK_VERSION
ac_add_options --enable-jemalloc
ac_add_options --enable-strip
ac_add_options --enable-devtools

ac_add_options --disable-eme
ac_add_options --disable-webrtc
ac_add_options --disable-gamepad
ac_add_options --disable-tests
ac_add_options --disable-debug
ac_add_options --disable-necko-wifi
ac_add_options --disable-updater

ac_add_options --with-pthreads

# Please see https://www.palemoon.org/redist.shtml for restrictions when using the official branding.
ac_add_options --enable-official-branding
export MOZILLA_OFFICIAL=1

ac_add_options --x-libraries=${lib.makeLibraryPath [ xorg.libX11 ]}

export MOZ_PKG_SPECIAL=gtk$_GTK_VERSION

#
# NixOS-specific adjustments
#

ac_add_options --prefix=$out

mk_add_options MOZ_MAKE_FLAGS="-j${if enableParallelBuilding then "$NIX_BUILD_CORES" else "1"}"
mk_add_options AUTOCONF=${autoconf213}/bin/autoconf

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Usefulness of development files in package

Unread post by Moonchild » 2021-05-03, 09:45

Your problem is you are using "mach install". You should not. This is not a standard "make && make install" simple Linux utility.
You should use "mach package" to prepare, strip, precompile and properly package the resulting browser. This will spit out a fully ready .xz file
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Re: Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 10:15

Is there no good way of avoiding the intermediary archive? That's just wasting time & resources on the build machines when we need to extract everything again.
strip
Our build system automatically strips the build results too, but only if the user doesn't request a debugging-capable result. If we switch to mach package, would there be an environment variable / argument to tell it to not strip its results or would users be SOL?

vannilla
Moon Magic practitioner
Moon Magic practitioner
Posts: 2183
Joined: 2018-05-05, 13:29

Re: Usefulness of development files in package

Unread post by vannilla » 2021-05-03, 10:44

To be fair the package-then-extract pass is not that terrible, even if it seems a waste of resources.
Compared to the actual building phase the time taken is irrelevant, plus you get some additional processing of some resources.
Basically, do it.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Usefulness of development files in package

Unread post by Moonchild » 2021-05-03, 11:24

OPNA2608 wrote:
2021-05-03, 10:15
That's just wasting time & resources on the build machines
I'm sorry but in what way is that wasting more time and resources than "installing" it to a temporary location? :think:

The main point is though: The packaging stage is simply an integral part of release engineering here. You should not skip it for any build. Period. You'd be hurting not only end user resources but also end-user performance. To understand why, have a look at the packaging stage in our tree and what all it does (I really don't feel like listing everything, sorry.)
OPNA2608 wrote:
2021-05-03, 10:15
would there be an environment variable / argument to tell it to not strip its results or would users be SOL?
You are already stripping as-you-build removing debugging bloat (since you're using --enable-strip) and unless you want positively huge binaries that is what you want. Basic debugging symbols are still included in the binaries; they aren't "bare".
What I meant with stripping in that context is excluding unnecessary files (strip the application/package) not related to debugging info.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Re: Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 13:06

I'm sorry but in what way is that wasting more time and resources than "installing" it to a temporary location? :think:

The main point is though: The packaging stage is simply an integral part of release engineering here. You should not skip it for any build. Period. You'd be hurting not only end user resources but also end-user performance. To understand why, have a look at the packaging stage in our tree and what all it does (I really don't feel like listing everything, sorry.)
We thought it's wasting time and space on paper since we're not interested in a tarball to distribute, we want everything unpacked to the output prefix. We do our own compressing from there if the build is running on a distribution server, to replicate the prefix 1:1 on the user's machine. The XZ archive way means we're going from build -> move to prefix (-> compress) to build -> compress to archive -> decompress to prefix (-> compress).

I understand now that this is the way your packaging is intended to work. It wasn't our intention to break your release engineering. I've been working on adjusting the build & install steps as we speak to instead use package and extract the archive contents where we need them to be.

I want to mention that stevepusser's Debian & Ubuntu build logs seem to suggest the use of install rather than package as well though.

E.g. in this recent build log I can find lines like make install and rm (those paths I initially asked about)

Code: Select all

[19449s] make -j1 -C /usr/src/packages/BUILD/obj-x86_64-pc-linux-gnu install
[19449s] make[3]: Entering directory '/usr/src/packages/BUILD/obj-x86_64-pc-linux-gnu'
[19449s] make[4]: Entering directory '/usr/src/packages/BUILD/obj-x86_64-pc-linux-gnu/palemoon/installer'
…
[19689s] rm -rf $(pwd)/debian/palemoon/usr/share/idl
[19689s] rm -rf $(pwd)/debian/palemoon/usr/lib/palemoon-devel
[19689s] rm -rf $(pwd)/debian/palemoon/usr/include
Which look similar to our logs with install.

Code: Select all

 0:01.36(B /nix/store/chy0dyh9imv0qj4x3xd1a2w4rw9v77pj-gnumake-4.3/bin/make -C . -j5 -s -w install
 0:01.36(B make: Entering directory '/build/source/obj-x86_64-pc-linux-gnu'
 0:01.40(B make[1]: Entering directory '/build/source/obj-x86_64-pc-linux-gnu/palemoon/installer'
And running package locally instead prints:

Code: Select all

 0:00.45(B /nix/store/y8i1aq9v5ibdg8lv6x370wc94p4hwq26-gnumake-4.3/bin/make -C . -j3 -s -w package
 0:00.46(B make: Entering directory '/build/source/obj-x86_64-pc-linux-gnu'
 0:00.51(B make[1]: Entering directory '/build/source/obj-x86_64-pc-linux-gnu/palemoon/installer'
…
 2:04.80(B make[2]: Leaving directory '/build/source/obj-x86_64-pc-linux-gnu/palemoon/installer'
 2:04.80(B Compressing...
 2:04.83(B palemoon/
…
With that Compressing... output completely missing from his logs.

Since those are considered fully-endorsed third-party builds by you, I assumed the package instruction was oriented more towards regular end-users and using install would be a good practice for us. I'm sorry if I'm misinterpreting any of those build logs though.
You are already stripping as-you-build removing debugging bloat (since you're using --enable-strip) and unless you want positively huge binaries that is what you want. Basic debugging symbols are still included in the binaries; they aren't "bare".
What I meant with stripping in that context is excluding unnecessary files (strip the application/package) not related to debugging info.
I see, apologies for the confusion. I wasn't thinking of file exclusion from a package, I'd usually associate that term more with binary symbols.
So leaving away --enable-strip would result in fat build artifacts? Would it be okay with you if I integrate that into our package argument system so users could easily toggle that and build a fully unstripped build with official branding - if they so desire - as long as we don't distribute such a build ourselves? The default would obviously still be a stripped one.

Our packaging system is very flexible and user-controllable. I'm working on reworking our current Pale Moon package into a generic base for source-built UXP browsers. My follow-up goal is a New Moon package with a (for better or for worse) fully user-controllable mozconfig, with our default mozconfig having sane-ish settings. After that will hopefully be a Basilisk / I-forgot-what-the-debranded-variant-was-called one. When I have the spare time for that. :lol:

New Tobin Paradigm

Re: Usefulness of development files in package

Unread post by New Tobin Paradigm » 2021-05-03, 14:34

Moonchild, you are incorrect. System packages should be created using the result of mach install which would get the build system to put files in standard (or chosen) linux locations (bin lib etc) then you capture the structure and files for the package to create the system package that follows the the linux filesystem hierarchy.

mach install WILL pass through mach stage step though if they are bypassing THAT with a hack then yeah they are still doing it wrong but otherwise it is incorrect to assume or direct system packagers to use mach package like we do for all-in-one distribution of gendric binaries.

See: http://xref.palemoon.org/moonchild-central/source/platform/toolkit/mozapps/installer/packager.mk#113

One thing I don't understand is why we are helping a hostile distro that has been mutually decided to not be official.
Last edited by New Tobin Paradigm on 2021-05-03, 14:48, edited 1 time in total.

User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Re: Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 14:47

One thing I don't understand is why we are helping a hostile distro that has been mutually decided to not be official.
I've never been hostile about the branding. The person you're talking about hasn't contributed to Nixpkgs in a long while AFAICT. I took over the packaging for Pale Moon more than a year ago, I verified with you that our mozconfig is fine for official branding, we kept the branding enabled and afaik it's been peaceful since :/…

New Tobin Paradigm

Re: Usefulness of development files in package

Unread post by New Tobin Paradigm » 2021-05-03, 14:49

Well I just woke up so mah brain may be fetching obsolete data. If so, sorry about that.

How do you clear your brain's cache again?
Last edited by New Tobin Paradigm on 2021-05-03, 14:50, edited 1 time in total.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Usefulness of development files in package

Unread post by Moonchild » 2021-05-03, 14:50

Well excuse me for not knowing what changed in Linux packaging then or how mach install is supposed to work because as far as I know precompiling startup cache, precompiling resource and creating .res and similar things is NOT done in "mach install", but I may be incorrect there too.

The files complained about at least seem to me files that should not be present in a properly staged binary. Manually copying icons should also not be needed. There should not be any -devel.* pr raw idl files anywhere in the compiled binaries; they don't belong there.

But since other people apparently know better, I'll just step aside. You figure it out.
New Tobin Paradigm wrote:
2021-05-03, 14:34
One thing I don't understand is why we are helping a hostile distro that has been mutually decided to not be official.
What are you talking about?
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Re: Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 14:54

Moonchild wrote:
2021-05-03, 14:50
What are you talking about?
https://github.com/NixOS/nixpkgs/pull/36917

New Tobin Paradigm

Re: Usefulness of development files in package

Unread post by New Tobin Paradigm » 2021-05-03, 14:58

Nothing has changed in that regard, Moonchild. It has always been that way. Just never really been discussed because we don't create system packages and it is something either understood already or simply ignored and the mach package form is Shanghai'd into a system package by those who just squeak by sort of like the binary repacks. Doesn't mean the mach package form is the proper method here.

Normally, we are JUST happy when they BUILD it properly. The packaging aspect isn't investigated until there is a reason. Now there is.

As an aside it makes no sense to export MOZ_PKG_SPECIAL outside of our generic binary with AUS production. This is something we use for distinction for our infra. It is in the DPMO linux and sunos mozconfigs for our common reference as much as the build instructions are for everyone else's.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Usefulness of development files in package

Unread post by Moonchild » 2021-05-03, 15:52

Well to be frank I never investigated the "install" target since... I just never had reason to.
Perhaps it needs improving. I'm certainly not the one suitable for it since I neither use *nix nor am I fluent in python or the packaging system.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
OPNA2608
Hobby Astronomer
Hobby Astronomer
Posts: 24
Joined: 2019-09-27, 09:30

Re: Usefulness of development files in package

Unread post by OPNA2608 » 2021-05-03, 16:02

New Tobin Paradigm wrote:
2021-05-03, 14:34
mach install WILL pass through mach stage step though if they are bypassing THAT with a hack then yeah they are still doing it wrong
AFAIK we're not hacking any build steps in the build process.

Our build server's logs: https://hydra.nixos.org/build/142322655#tabs-summary (press pretty or raw next to Logfile:)

Our complete packaging code: https://raw.githubusercontent.com/NixOS ... efault.nix

I'm trying to be very open about my packaging effort with you. Anything that isn't self-explanatory Nix functions or visible shell code:
  • Order of everything visible: src is unpacked & entered -> configurePhase -> buildPhase -> installPhase -> preFixup (part of fixupPhase)
  • Any instances of runHook are out of style convention and don't execute any code, afaik.
  • Our current compiler is GCC10, I had pinned GCC8 for Pale Moon previously because your instructions list it as the last not-experimental compiler but this broke GL support on my new GPU due to the OpenGL library loading getting all broken with the different GCC versions. (¯\_(ツ)_/¯)
  • Our compilers & stdenv are written to catch illegal paths in our build environment, deal with cross-compilation & architecture emulation and insert hardening-related flags. I mentioned the specific flags we use in viewtopic.php?f=37&t=24668#p192895 and they seemed to be okay with you.
  • The preFixup phase, along with some automatic GTK-specific environment variables (GDK_PIXBUF_MODULE_FILE, GDK_PIXBUF_MODULE_FILE, XDG_DATA_DIRS to gsettings schemas), wraps ffmpeg and pulseaudio library paths into the LD_LIBRARY_PATH of the PM binary so it can find these optional libraries at runtime. Our distro doesn't follow the Linux FHS so this is needed (alternative would be ELF modifications via patchelf, either extending the rpaths or adding the libraries as fixed load-time dependencies).
  • In the inherited fixupPhase all libraries & binaries in the installed prefix are shrunken (rpaths reduced to the minimum required for the library dependencies) and stripped, and shebangs that assume a Linux FHS are rewritten to work in Nixpkgs
I've used this package on NixOS and Ubuntu for the last 1½ years and it's worked very well for me, minor hickups aside. :D
New Tobin Paradigm wrote:
2021-05-03, 14:58
As an aside it makes no sense to export MOZ_PKG_SPECIAL outside of our generic binary with AUS production. This is something we use for distinction for our infra. It is in the DPMO linux and sunos mozconfigs for our common reference as much as the build instructions are for everyone else's.
I'll remove it if you say it's not required for us, I stuck to your build instructions for the most part.

All of that aside do I understand correctly that it's fine for us to stick with install and delete those paths for now?

New Tobin Paradigm

Re: Usefulness of development files in package

Unread post by New Tobin Paradigm » 2021-05-03, 16:09

Well there isn't MUCH benefit from installing the XPCOM SDK (it can be used from dist just fine if needed) from most linux perspectives. Could duplicate the target as a different name then change the original to exclude the cruft or just split them into two different targets. So that makes sense to me Moonchild.

As for you OPNA2608, yeah just exclude it from your system packaging.

Locked