Build system status

Discussions about the development and maturation of the platform code (UXP).
Warning: may contain highly-technical topics.

Moderators: trava90, athenian200

User avatar
micwoj92
Fanatic
Fanatic
Posts: 174
Joined: 2020-12-22, 20:57

Build system status

Unread post by micwoj92 » 2022-09-23, 18:17

"follow up" to https://repo.palemoon.org/MoonchildProd ... ssues/1878

Today python2 has been dropped from main Arch repos to AUR, got me wondering if there has been any behind the scenes effort or thoughts on this.

Mozilla (and other forks of -zilla based software) still are using autoconf2.13 so I doubt it will be problem anytime soon.

From related projects (other than mozilla sofware itself) I know that seamonkey has been working on porting their build system to python3 https://bugzilla.mozilla.org/show_bug.cgi?id=1756371 Hope this is of any help.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Build system status

Unread post by Moonchild » 2022-09-23, 19:06

We have no interest in breaking our entire build system because python 2.7 is no longer default installed on some distros. If you want to build Pale Moon, just make sure you install Python 2.7, autoconf 2.13 and other prerequisites. Yes, that means you need to put some time into setting up your build environment. No, that's nothing strange from a developer's point of view.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1480
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Build system status

Unread post by athenian200 » 2022-09-23, 23:42

Well, back when it was being discussed a few years ago, I believe we were discussing potentially using Tauthon if Python 2.7 is no longer an option or gets to a point where it can't be compiled on most distros. That's a fork of Python 2.7 that is still being maintained. I don't believe we are extremely dependent on the distro being able to fetch the Python libraries it needs from living servers at runtime, and I think we have all the Python libraries we depend either in our tree or shipped with Python 2.7. So we really only need an interpreter that can run it all.

My attitude on it is, if someone can actually port our build system over to Python 3.x and verify to our satisfaction that nothing was broken in the process, then I would be fine with it. The problem is... well if you look in our codebase, you see that our Python-based build system is huge and complicated. It's almost as bad as the migration Gentoo had to make with Portage, I think. They actually had help with that, and I believe it took years and was fairly piecemeal. In other words, we would likely have to have a long period of time where the codebase required both Python 3 and Python 2 to build, as modules were migrated over in stages. That's how I believe Mozilla and most of the other organizations saddled with Python-based build systems handled things.

From what I've been told, Python 3.x is so different from 2.x that the majority of existing code would simply not work, and it would be like porting it all over to an entirely new language. And at that point, why should we trust that they won't push Python 4.x while we're in the middle of migrating? If Python is just going to radically change every few years, and distros will in turn demand everyone adapt their code to those radical changes or they are not allowed to run, that's not going to work for us. The other problem is... well, I'm not that familar with Python in the first place, and I don't know whether MC is either. And a migration like this would likely require a Python specialist who could pretty much do this in their sleep, preferably also one with knowledge of the mach build system.
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Build system status

Unread post by Moonchild » 2022-09-24, 01:13

athenian200 wrote:
2022-09-23, 23:42
I believe we were discussing potentially using Tauthon if Python 2.7 is no longer an option or gets to a point where it can't be compiled on most distros. That's a fork of Python 2.7 that is still being maintained.
It's effectively the same, as far as I understood. Sure it's nice if Tauthon is more easily accessible but it would not mean we are changing our python compatibility -- something Python 3.x does require.
athenian200 wrote:
2022-09-23, 23:42
if someone can actually port our build system over to Python 3.x and verify to our satisfaction that nothing was broken in the process, then I would be fine with it.
I just think it's really not worth the trouble. Python 2.7 will be available one way or another. If a company the size of Mozilla stuck with autoconf 2.13 for literally forever because its behaviour was so integral to the build system, instead of rewriting every build system script for the next version, I'm sure you can see the analogy here.
I don't think Python 3 is a feasible consideration to make.
athenian200 wrote:
2022-09-23, 23:42
And a migration like this would likely require a Python specialist who could pretty much do this in their sleep, preferably also one with knowledge of the mach build system.
I don't think any such person exists. And if they do, they are likely already employed at Mozilla ;-) (and probably outside of our project's budget to hire)
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
andyprough
Astronaut
Astronaut
Posts: 688
Joined: 2020-05-31, 04:33

Re: Build system status

Unread post by andyprough » 2022-09-24, 02:51

Moonchild wrote:
2022-09-24, 01:13
athenian200 wrote:
2022-09-23, 23:42
I believe we were discussing potentially using Tauthon if Python 2.7 is no longer an option or gets to a point where it can't be compiled on most distros. That's a fork of Python 2.7 that is still being maintained.
It's effectively the same, as far as I understood. Sure it's nice if Tauthon is more easily accessible but it would not mean we are changing our python compatibility -- something Python 3.x does require.
I've built Pale Moon on Hyperbola GNU/Linux with Tauthon. Hyperbola does not offer Python, but uses Tauthon instead. I did not notice any difference, and the build process went smoothly and the browser worked fine. In fact, I've built a few different versions that way.

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1480
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Build system status

Unread post by athenian200 » 2022-09-24, 05:18

andyprough wrote:
2022-09-24, 02:51
I've built Pale Moon on Hyperbola GNU/Linux with Tauthon. Hyperbola does not offer Python, but uses Tauthon instead. I did not notice any difference, and the build process went smoothly and the browser worked fine. In fact, I've built a few different versions that way.
Yeah, honestly I prefer Tauthon over Python at this point, because I like the idea behind what they're doing... trying to not break older code while adding useful features in an evolutionary way, which in my view is what programming languages should do rather than radical redesigns that break everything, change the rules, and move everything around for very little gain.
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Build system status

Unread post by Moonchild » 2022-09-24, 08:23

I agree with evolution, not revolution, when it comes to programming languages. And I think that that evolution should be entirely developer-driven, too, meaning things everyone agrees to would be beneficial to the language and its paradigms are the only ones that actually get added or changed. I've always been of the opinion that if you want syntactic sugar, for example, that that is exactly why you have libraries.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

dbsoft
Project Contributor
Project Contributor
Posts: 405
Joined: 2020-02-21, 17:35

Re: Build system status

Unread post by dbsoft » 2022-09-27, 18:01

I'm generally in agreement here, while I am not really a fan of Python as a language generally, Tauthon seems like a better evolutionary step, I think the breaking changes of Python 3 were a mistake. My concern is just that Tauthon isn't as accessible as Python is, and worry at some point even Tauthon will fall by the wayside. Ideally upgrading everything to Python 3 would be desirable... or for me some other language that I don't dislike. I surely don't have an interest in taking that on, and don't have the necessary Python skills anyway. For the time being if Python 2.7 isn't available, Tauthon should work. If someone with Python skills volunteers to port everything to Python 3 I would surely welcome that though.

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1480
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Build system status

Unread post by athenian200 » 2022-09-27, 19:46

dbsoft wrote:
2022-09-27, 18:01
I'm generally in agreement here, while I am not really a fan of Python as a language generally, Tauthon seems like a better evolutionary step, I think the breaking changes of Python 3 were a mistake. My concern is just that Tauthon isn't as accessible as Python is, and worry at some point even Tauthon will fall by the wayside. Ideally upgrading everything to Python 3 would be desirable... or for me some other language that I don't dislike. I surely don't have an interest in taking that on, and don't have the necessary Python skills anyway. For the time being if Python 2.7 isn't available, Tauthon should work. If someone with Python skills volunteers to port everything to Python 3 I would surely welcome that though.
Agreed. It won't be as big a problem on Windows since MozillaBuild can be pretty much frozen in time as the environment we build things in, but on Linux people have an expectation that they will be able to build with the tools that come with their distro, and that to some extent developers will keep up with major packages becoming deprecated. Of course, the "rules" are bent for big companies like Mozilla who can pull strings and get something like autoconf 2.13 to be included forever.

I mean, there are workarounds if it comes down to it and this becomes a bigger problem. Aside from the obvious solution of simply building Tauthon or Python 2.x from source (maybe using AUR or some kind of community-maintained overlay for Gentoo). People could build inside distrobox or a Docker container with an older distro, maybe someone can maintain a developer repo where people can get Tauthon binaries for the more popular distros that decide to drop it, or maybe a statically linked copy that should work on almost any version of Linux. Supposedly there are also some Python interpreters that will be sticking with 2.x that aren't based on CPython, but our build system depends on CPython so it would be work to adapt there as well. The point is, there are workarounds, and almost all of them are probably easier than porting MozBuild to Python 3.x.

It just kinda sucks that it will probably never happen, because it means we will naturally have fewer people able to build on Linux over time, and our universal binaries don't work quite right on every distro, so there will inevitably be people on Linux who can't build Pale Moon themselves easily, and who find that the binaries we provide don't work for them either. Sure, having distro maintainers for each distro is ideal, but a lot of distros don't want Pale Moon in their repos, and that just kinda... is what it is, you know? It's just too big a job and there's not a clear advantage to something like this, as there would be to stuff like compiler and linker updates over time. I mean, the Python-based build system isn't really a part of Pale Moon proper... it's just machinery, and there is so much more pressing stuff that can be done with programming languages we are more comfortable with like C++ and JavaScript.
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
micwoj92
Fanatic
Fanatic
Posts: 174
Joined: 2020-12-22, 20:57

Re: Build system status

Unread post by micwoj92 » 2022-09-27, 20:26

athenian200 wrote:
2022-09-27, 19:46
but on Linux people have an expectation that they will be able to build with the tools that come with their distro, and that to some extent developers will keep up with major packages becoming deprecated.
Yes, that was my main point
athenian200 wrote:
2022-09-27, 19:46
Of course, the "rules" are bent for big companies like Mozilla who can pull strings and get something like autoconf 2.13 to be included forever.
I will say that there is also some difference between entire programming language and "just a couple scripts"
athenian200 wrote:
2022-09-27, 19:46
Aside from the obvious solution of simply building Tauthon or Python 2.x from source (maybe using AUR or some kind of community-maintained overlay for Gentoo).
I maintain python2 on aur now and probably will until it won't cause too much problem. I'm currently looking for random python2 packages, checking if they work with python3 and updating/messaging maintainers.
athenian200 wrote:
2022-09-27, 19:46
Supposedly there are also some Python interpreters that will be sticking with 2.x that aren't based on CPython, but our build system depends on CPython so it would be work to adapt there as well. The point is, there are workarounds, and almost all of them are probably easier than porting MozBuild to Python 3.x.
I tried with pypy (as it is still in Arch repos), but it was too much hassle. I think it the end it would still be less work to port to python3 rather than support (possibly few) python2 implementations. It would then probably only require once a year recheck for new python3 versions. If Seamonkey can do it then why can't Pale Moon, sadly complicated build systems require "a bit" more than running 2to3 and hoping it works
athenian200 wrote:
2022-09-27, 19:46
Sure, having distro maintainers for each distro is ideal, but a lot of distros don't want Pale Moon in their repos, and that just kinda.
Imo it's mostly due to hatred (of you know who) towards "illegal" redistributions in things like OpenBSD. License is not ideal as you could even see that in Mozilla's example with their dispute with Debian where there was Iceweassel for some time. So licensing and pr plays a big role. Also most of the time Pale Moon doesn't "add" much to a distro, who knows, maybe after manifest v3 will finally take effect people will move to alternatives and there will be some influx of devs on Pale Moon.

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1480
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Build system status

Unread post by athenian200 » 2022-09-27, 21:05

micwoj92 wrote:
2022-09-27, 20:26
I tried with pypy (as it is still in Arch repos), but it was too much hassle. I think it the end it would still be less work to port to python3 rather than support (possibly few) python2 implementations. It would then probably only require once a year recheck for new python3 versions. If Seamonkey can do it then why can't Pale Moon, sadly complicated build systems require "a bit" more than running 2to3 and hoping it works
Well, for someone who is familiar with both Python 2 and Python 3, they could probably figure it out over time. The thing about SeaMonkey is, they stayed closer to Mozilla than we did, and I think the version of the build system they inherited already mostly supported Python 3. I think additionally, they also have the help of people who actually know Python 3.x and have experience with migrations. On our side, we have no real offers of help, and also parts of the build system that Mozilla never updated for Python 3 that we still need. So while we could probably take some of what we need from SeaMonkey and Mozilla, I'm not sure it would be enough to port over the pieces they never updated to Python 3 that we still need, like the XPIDL stuff, etc.

The other thing is... most of the main people working on this codebase don't really understand Python beyond what we've learned tinkering around with the build system, and most of that is 2.x. That is to say, we effectively have no experience with Python 3.x and only know a little about Python 2.x in the context of the build system. For instance, I don't even know what "2to3" is. It sounds like some kind of migration tool or service? From our perspective, no Python 2.x modules are going into the program itself, it's just something (unfortunately) needed to build the program, same as autoconf and GNU make before it. We don't really need Python itself, we just need a mess of scripts to work that happen to have been written with Python, that could have just as easily been written in Perl or something.

It's just for whatever reason, the PSF unfortunately is the kind of organization that expects people to "just update" to something that is almost completely different to the point of being a new language with a few similarities. I'm also kind of worried that by the time we did migrate to Python 3.x, they'd come up with Python 4.x,and it would just be a cycle that never ends where we always have to put time into keeping up with their nonsense. I don't trust them on API stability, and it's hard enough to keep up with constantly evolving web standards without having to deal with the Python people having the same "move fast and break things" mentality where they don't care if your old programs work or not and just have this mentality that if you can't keep up with their random changes to how the language works you "don't deserve to survive." Python is almost as bad as Rust in terms of constantly changing the API (except at least it isn't code that goes into the final program).
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Build system status

Unread post by Moonchild » 2022-09-28, 00:46

athenian200 wrote:
2022-09-27, 21:05
it would just be a cycle that never ends where we always have to put time into keeping up with their nonsense
And unlike them, we can't dedicate 100% of our time to fixing python things, because we have other, more important things to do than rewriting our entire build system constantly. We use it as a tool, and our main work has little to nothing to do with python development.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35402
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Build system status

Unread post by Moonchild » 2022-09-28, 00:54

Off-topic:
micwoj92 wrote:
2022-09-27, 20:26
Imo it's mostly due to hatred (of you know who) towards "illegal" redistributions in things like OpenBSD.
IMO that was solved years ago already. It just seems that people have trouble letting go of something once it has escalated even just once. Then again, some people make it their mission to "keep it alive" because it's being treated as some Great Injustice That Shall Not Be Forgotten™ no matter the clear explanations I myself have also given on the different matters dealing with problematic build configurations that really, ultimately, produce something that at first glance may look like Pale Moon, but technically isn't. And in that case it should not carry my branding. It goes back to the roots of the GNU philosopy and free software.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked