A cheap trick to reduce memory usage building a UXP application (such as Pale Moon)

[The Squash]

The problem

So... you've decided, for one reason or another, that you need to build a UXP application. Maybe you are testing out somebody's new UXP-based application, for which no binaries yet exist. Maybe you intend to run a supported application on an unsupported computer architecture. Maybe you have very specific needs for the application. Or maybe you're just a Gentoo freak... (I sympathize with all four categories!)

But there's one problem: You have hardware which is mostly capable of compiling the application, but falls juuust a little short of compiling the whole thing. Or maybe you are building the application on a 32-bit system. Whatever the reason, the result is the same: The compilation of UXP almost completes, but a major failure occurs in the linking process.

So now (especially considering that old and slow hardware) you've just blown 3 hours compiling UXP for nothing. Grrr! Once you regain your sanity, read this article; it may just help you.

The solution

Chances are, if you're compiling an application for yourself, you probably are not compiling it for debugging per se; you just have other special needs. If you are compiling it for debugging, then stop reading here, because this howto does not cover those bases.

Still here? Okay, now we can talk about the solution.

The solution is to "strip" the unneeded symbols out of the compiled but unlinked code -- known as object code -- before the main UXP library is linked. You see, all of the object files UXP is composed of may add up to over 2 GB of disk space. The linker needs to read all of those files into memory and produce a combined, linked library from those separate files. Normally, the "stripping" of unneeded parts of the object files occurs after the linking is done. But my hack will show you how to do this before the linking is done, to save memory and speed up the linking process too. In my tests, stripping the objects files prior to linking reduced the disk usage of the object files by a factor of 10.

Method #1: If you've already gotten most of the way through a build

If you've already gotten most of the way through a build, then the process is easy, at least on UNIX-like operating systems. If you have a terminal open and the shell is in the directory where the application is located, you can run the following one-liner:

Code: Select all

find obj-* -name '*.o' -type f -exec strip --strip-unneeded {} \;

This tracks down all object files ('*.o') which are regular files (-type f) and then strips unneeded symbols from them.

WARNING! Do not use strip on the object files without any command line switch! If you do, the linking process will fail and you will have to start all over!

Once this process is done (it should take a few seconds), rerun the build and see if the linking succeeds this time.

Method #2: If you're just starting out a build

If you're just starting out a build, nothing could be easier. Just open the file 'platform/config/rules.mk' under the application's source tree, using your favorite text editor. When it opens, scroll down to the section which reads:

Code: Select all

ifndef XP_MACOSX
	dtrace -x nolibs -G -C -s $(MOZILLA_DTRACE_SRC) -o  $(DTRACE_PROBE_OBJ) $(shell $(EXPAND_LIBS) $(MOZILLA_PROBE_LIBS))
endif

Right after the 'endif', insert a newline and add the following (note: the indentation is two tabs, not 16 spaces!):

Code: Select all

ifdef ENABLE_STRIP
ifneq ($(strip $(STATIC_LIBS)),)
		$(EXPAND_LIBS_EXEC) -- $(STRIP) --strip-unneeded $(STATIC_LIBS)
endif # $(STATIC_LIBS) != ""
endif # ENABLE_STRIP

Then, a few lines later in that file you should see the lines:

Code: Select all

	$(EXPAND_MKSHLIB) $(SHLIB_LDSTARTFILE) $(OBJS) $(SUB_SHLOBJS) $(DTRACE_PROBE_OBJ) $(MOZILLA_PROBE_LIBS) $(RESFILE) $(LDFLAGS) $(WRAP_LDFLAGS) $(STATIC_LIBS) $(RUST_STATIC_LIB_FOR_SHARED_LIB) $(SHARED_LIBS) $(EXTRA_DSO_LDOPTS) $(MOZ_GLUE_LDFLAGS) $(EXTRA_LIBS) $(OS_LIBS) $(SHLIB_LDENDFILE)
	@$(RM) $(DTRACE_PROBE_OBJ)
else # ! DTRACE_LIB_DEPENDENT

Add the following lines after the 'else' line:

Code: Select all

ifdef ENABLE_STRIP
ifneq ($(strip $(STATIC_LIBS)),)
		$(EXPAND_LIBS_EXEC) -- $(STRIP) --strip-unneeded $(STATIC_LIBS)
endif # $(STATIC_LIBS) != ""
endif # ENABLE_STRIP

Save the file and close your editor.

Now start the build process as usual and you should be good to go!

[New Tobin Paradigm]

Do not do any of this. This person has no idea what they are talking about.

[athenian200]

But what if it strips out a symbol that wasn't needed when the object itself was linked, but then the removed symbol is later called from another object file that object was linked with? You can't know before linking what symbols might be called from other objects it gets linked with, so the stripping is premature.

[Moonchild]

But there's one problem: You have hardware which is mostly capable of compiling the application, but falls juuust a little short of compiling the whole thing. Or maybe you are building the application on a 32-bit system.

The you should re-think your computing choices. If your machine is unable to compile and link normally in your environment, then your environment is ill-suited to be building it and you simply shouldn't do it.

[The Squash]

You can't know before linking what symbols might be called from other objects it gets linked with, so the stripping is premature.

That's not true. If you strip all symbols, sure; but my method only strips non-global symbols. Local, non-exported symbols simply shouldn't be used outside the libraries. If they are, the library has a bug.

This mechanism actually works, BTW.

[New Tobin Paradigm]

You are out of order. This thread should be moved out of this board to somewhere else because nothing this person says is valid. Must be Dnes's long lost brother.

[Moonchild]

Topic moved.

I'll say it once more:
The Mozilla code tree is not your standard application tree and does use a lot of (often dirty) tricks to glue everything together, including using implied symbols and variable types.
You may think this works but you may also (likely) be creating issues re: operation and stability (and possibly security if you're creating pointers to non-existent references as a result).
Did you actually test and verify that you're not creating any of these kinds of issues?

Once again, if you can't properly build this as-is, then you should not be building it, period, and you should be using a prebuilt binary OR upgrade your build system to have the minimum required memory and OS.

[The Squash]

if you can't properly build this as-is, then you should not be building it, period

Isn't one of the rules about building UXP applications that you do so at your own risk? I thought that rule was implied.

Also I think I tested it pretty thoroughly. I ran Valgrind on libxul and in fact I'm using the fruits of this hack right now. I've actually encountered no crashes lately, which is extremely odd since I usually encounter at least one crash per official version I use (before I update Pale Moon to the latest version available). I compiled this thing on August 6, and it's run flawlessly on hundreds of websites so far.

BTW, I do realize that 28.13.0 is out and that I'm not supposed to talk about 28.12.0 anymore. I would be using 28.13.0 right now but I'm trying to prove a point.

BTW, I suspect the crashes with the official version are because it's compiled on CentOS and CentOS is patched beyond repair -- the packages in Gentoo are identical (for all practical purposes) to the upstream packages.

[New Tobin Paradigm]

Acting as if you are an authority suggesting ill advised or dangerous actions in configuring, operation, or building simply isn't going to be tollerated certainly not allowed to pass without harsh comment.

We have been doing this for a long time now and your history here has shown how you have progressed from trying to use our community and infrastructure to recruit people to bring your non-existant gtk fork out of vaporware into reality, to aggressively trying to back seat moderate this forum when your discussions or discussions you jump into don't go your way, to this Dnes-level shit regarding build actions except that while Dnes is an irritant and mostly harmess you are suggesting dangerous actions as a tutorial in order to present the illusion that you know what you are talking about all in the guise of helping.

This makes you a clear and present threat to the userbase because most of them don't know any better but they do have one trait I will not disparage them on: trust that those here that do work on this code or are engaged in technological research and discussions aren't out to nor will allow any one actor to actively try and fuck them over.

This is where we are now. You are actively either through sheer entitled ignorance of reality or by malicious intent trying to fuck other people over with demonstratively dangerous instructions. You must be stopped and exposed before someone does what you suggest and gets burned for it.

[The Squash]

I know plenty of people reading this thread right now (and in the prior hours) who are laughing.

Why?

Because these people (I shall not name them -- all requested not to be named) have in parallel with me tested this suggestion and none of them nor I have encountered any issues yet, after over a month of testing.

Because the fact is, old versions of the strip command actually used to cause harm. But nowadays the strip command has become a lot smarter about what to remove and what not to remove.

Nevertheless, if you do not believe this is advice to be taken seriously, then you do not need to use it.

By the way, has anyone here speaking yet tried this method?

[New Tobin Paradigm]

So, you are acting as the front man for a group eh? What group? Who are you and who are your accomplices? More importantly, why are you here?

[Moonchild]

How about we all take this as what it is: a cheap trick, as stated in the topic.

For all the above reasons I advise against it, no matter how good you think strip is on Gentoo (because I'm assuming that is what you are fronting for here, the group who has complained loudly about resource requirement having increased for building). if My assumption is wrong then please, by all means, enlighten us who this "group of people" is you are speaking for.
And I also advise strongly against building on underpowered machines with insufficient address space to normally build the tree.

Why do we give resistance? because people who would take this the way you posted it (as a howto/tutorial, and then defended with claims, i.e.: as if this would be in any way within scope which it is not) will either come to us with instability issues or (worse) just get a terrible impression of the code and will advise anyone to use it in any way, damaging all current and future UXP projects and applications. Just because you can doesn't mean you should.

All that being said, I reiterate once more that you are, absolutely, on your own with hacking the build system to "make it work" in ways it is not designed and which our decade experience red-flags immediately. Just make absolutely clear to whatever group you are representing that it is very dangerous what you are doing and you can easily cause massive security issues too.

And just in case you missed it the first time:

Doing this is not within scope of our source's usage.

[The Squash]

Should the people in my group choose to speak out, they shall be allowed to do so.

In the meantime I've promised not to drag any of their names into the mix, so unfortunately you can't know who's involved.

[Lurker_01]

Fails on linking XUL with "io: not enough space" error on Windows (with free space available on disk) in my case, making this a no go change.
Link still maxes out on memory usage as it should, after reverting this only for the last step, so the trick is useless and harmful.
I have a machine beefy enough for normal build process, so i wont use this for my own personal build.

Off-topic:
I would never report bugs without testing on official build and my build is not distributed. adding this disclaimer here before someone would jump to conclusions.

[Tharthan]

Off-topic:
A trustworthy source has informed me that The Squash is actually the front man for underground moblins.

And, no, I cannot reveal my source to you, The Squash. It's a secret to everybody.

[athenian200]

Should the people in my group choose to speak out, they shall be allowed to do so.

In the meantime I've promised not to drag any of their names into the mix, so unfortunately you can't know who's involved.

Well, the one person who has admitted to trying your solution is saying it doesn't work well on Windows. Also, the stuff you're saying about having followers sounds a bit like: "Better be careful how you treat me, I have an army! You can't see them because they're all camouflaged. But trust me, I have an army. "

Your friends notwithstanding, though, it seems like your efforts would be better spent on looking for a concrete way to demonstrate that on a given system, there's not going to be a meaningful difference between the final binaries produced this way, and the final binaries produced the official way. If you can't demonstrate that, then that's a problem. You may think your approach is theoretically sound, but you have no way of knowing what kind of bugs exist in the codebase or whether a non-standard approach is taken somewhere with how symbols are used, etc.

[The Squash]

Link still maxes out on memory usage as it should

Did you pass '--enable-strip' to the build process? Do you have GNU strip installed?

[New Tobin Paradigm]

I agree with others here regarding the implication of a threat being expressed. Know this: I will not give in to terrorism nor will I allow any of the others to do so without serious question.

You and your hypothetical group cannot hope to stand against me myself in the worst of situations so what makes you think you can succeed when others also see through your attempt at posturing and general ignorance.

[Lurker_01]

Did you pass '--enable-strip' to the build process?

Yes i did have '--enable-strip' in the .mozconfig and it shows that it works, otherwise it wouldn't have error-ed out if it didn't enter the new code-path.

Do you have GNU strip installed?

Okay i think you hit the problem here.
I have all pre-requisites listed on http://developer.palemoon.org/build/windows/
strip is not included in the MSYS of MozillaBuild 2.2
I have downloaded binutils for MinGW and added them to MSYS and checked that i can call strip from the console.
Then i ran it as a test with your standalone command to see if it works fine and found out the obvious thing that using gnu strip, to strip non gnu c++ object files wont work... i modified the command to find .obj files of MSVC and the result is "file format not recognized". (ran the build anyway and got the same result)

So this is a bust for Windows (checked that there is a lot of questions about 'equivalent of the GNU "strip" tool for Windows' in the web with negative or no responses) and so you must add an OS check in your modifications.
I am sorry, i wont test this on Linux as setting up a full Linux build environment even on an VM when i don't have it at hand would take too much time.

[Lunokhod]

The problem with testing is that you only test your build on your system. The official Pale Moon builds are intended to work on all systems. Any build will depend on the build environment as things like installed library, kernel and compiler versions will build different binaries. Also there are loads of build options. AFAIK the complete Pale Moon build recipe has never been revealed, nor any testing schedule. Specifying 1 job in the mozconfig would probably cut resources. It's an interesting trick for general use on big projects, or to cut build time if you were only testing some detail rather than building something to actually use. Unless you have well spec'd HW it's going to take ages to build anyway.
Who is The Squash indeed? I'd wondered at a Mint / Ubuntu link as there seems some enthusiasm expressed about STLWRT from those quarters. So now there are The Squashes too, who as yet remain silent.