Build system development (python3, perl, etc.)

[Moonchild]

I'd actually be quite interested in the challenge of rewriting mach to use Python 3 - for context, I use Python in my career and regularly do large-scale refactors, so although I'm sure it'd be a big job, I don't think it would be unachievable. However, I'm not sure I'd want to start looking into it without at least some indication that my rewrite might be used if successful.

I'd welcome such an effort, for sure, but it goes way beyond "mach". Our entire build system, build file parser, various generator programs, etc. etc. are all written in python and built on python 2.7 support only. Mozilla has a few meta bugs that should give you an idea of the scope of this, e.g: bug #1496527 bug #1388447 and bug #1473499.
However, Mozilla made things convoluted by supporting python 2 and 3 in parallel, and that's not really something I think we need to do, which would probably greatly simplify it.
The reason why we haven't done anything like this so far is primarily because we've not had the python knowledge in our ranks to do so.

[Keiji]

I'd welcome such an effort, for sure,
[...]
The reason why we haven't done anything like this so far is primarily because we've not had the python knowledge in our ranks to do so.

Thanks for the support! I wasn't sure if it was a case of "we want to do this, but there's no one to do it" or whether there was some other reason not to move to Python 3. I'd be happy to work on this!

but it goes way beyond "mach". Our entire build system, build file parser, various generator programs, etc. etc. are all written in python and built on python 2.7 support only. Mozilla has a few meta bugs that should give you an idea of the scope of this, e.g: bug #1496527 bug #1388447 and bug #1473499.
However, Mozilla made things convoluted by supporting python 2 and 3 in parallel, and that's not really something I think we need to do, which would probably greatly simplify it.

Thanks for the context - I've had a quick look over it, these are certainly useful references. I plan to have a more thorough look through the actual .py files a bit later on, and get a rough idea of the scale involved.
How much (if any) has the Python code in Pale Moon changed from Mozilla's, since the original fork?
Just going by your gut feel, do you think it would be more efficient overall to study Mozilla's porting effort and attempt to replicate it (or even cherry-pick if we're lucky), or to completely ignore their port and do the job from scratch, considering PM's code only (especially as we don't need to bother with a transition period, as you mention)? I'm the kind of person that's tempted to do the latter, but if there's some pieces that clearly ought to be applied as-is first, this would be good to know about.
I may have the Python experience, but you have the experience in trawling through Mozilla's changes - that's certainly not something I've ever done much of myself!

Coincidentally, it looks like Mozilla is moving from Mercurial to Git - as an experienced Git user (and not a Mercurial user at all), if I'm going to be studying any Mozilla history, I might hold fire on that til they make their public Git repo (assuming it's going to contain the history and not just be a snapshot used moving forward, of course).

Off-topic:
By the way, would you mind splitting this to a new topic back in the development forum? I have a feeling this is going to get quite involved, and I'd still like this topic to stay as a how-to on building with Python 2. Perhaps my own fault for making a PS rather than a new topic in the first place, though...

[athenian200]

PS. I'd actually be quite interested in the challenge of rewriting mach to use Python 3 - for context, I use Python in my career and regularly do large-scale refactors, so although I'm sure it'd be a big job, I don't think it would be unachievable. However, I'm not sure I'd want to start looking into it without at least some indication that my rewrite might be used if successful. I'd ultimately like to see more adoption of Pale Moon, and while I don't know the exact reasons why Linux distributions do not provide packages for it, I imagine the whole Python 2.7 issue is probably one big reason. (I could also test any new mach on Win10 & 11 not just Linux, and heck I've been wanting to give BSD a try too, though I wouldn't be able to test macOS support at all.)

I've actually been considering rewriting the build system in Perl 5, since I am a lot more familiar with Perl. Also, I was considering going forward with a plan that was put forward at one point to at least get rid of mozconfigure in favor of going back to autoconf and m4, even if we probably do need to keep mozbuild and possibly convert that over to Perl or something eventually (building UXP already requires Perl, so it wouldn't change build requirements). Originally, going back to shell scripts was also part of the plan, but I've soured on that part of it, simply because I've worked with Solaris, various flavors of Linux, and even seen a few people use BSD, and found that supposedly simple shell scripts are rarely as cross-platform as they seem, but that stuff like Python and Perl does work a lot better as a cross-platform scripting language. Perl in particular is fairly good at text processing and has a more C and shell like syntax, while Python is totally alien to me.

My thinking is, if Python 3 is a brand new language the Python devs just expect everyone to move over to, and I only have limited knowledge of Python 2, but know Perl reasonably well, then I might as well rewrite everything in Perl instead of Python, right? I mean, what guarantee do we have that the moment we get things converted over to Python 3, they won't pull the rug from under us again and decide to create another new language called Python 4 and expect us to learn that one? Python seems to be very much a "move fast and break things" kind of language that doesn't care about backwards compatibility, and that just makes me feel like if we have a choice, it's really not suitable for our purposes because everyone in the Python community from the main devs to the package maintainers for individual distros expects you to be willing to rewrite your scripts in a brand new programming language at the drop of a hat.

But yeah, if you can do Python 3, that would probably be easier for everyone, and keep us more in line with Mozilla so we could use more of their stuff. But in general, the challenge is that we don't necessarily want to follow Mozilla on everything, like we don't want to get rid of m4 or autoconf, etc. But really, none of this has been a priority for us because Python 2 works well enough, the Linux distros have just soured on it and bought into the finger-wagging "it's obsolete, you must upgrade" attitude on this, not having much respect for the fact that it's a totally new language and people have years worth of scripts written in it.

[Moonchild]

the Linux distros have just soured on it and bought into the finger-wagging "it's obsolete, you must upgrade" attitude on this, not having much respect for the fact that it's a totally new language

Actually, a lot of them, the vast majority, don't seem to understand that a build and dev environment isn't necessarily the same as a normal use environment. e.g. python 2.7 may be "insecure" but it will not be if it's only being used with known and trusted scripts. There's a big divide between using tools with known scripting or using it with untrusted/foreign/third-party scripting.

[Keiji]

@athenian200:

FWIW, I don't really agree with your analysis.

Yes, Python 3 broke many things, but it's far from "a brand new language". They broke BC for strings and for a fair bit of commonly-used syntax, notably exceptions, but it's nowhere near the level of difference between, say, C and Rust, or C and PHP, or even C and C++. It's also worth mentioning that the first few releases of Python 3 were... bad. IIRC it was around 3.3 or 3.4 when Python 3 actually became worth it, so if you're basing your viewpoint on your experiences around that time or earlier, I can understand the dislike of Python 3 (I myself didn't want to upgrade back then either). For comparison, I recently upgraded a decently sized project from 3.7 to 3.10, and it really was not that hard, other than one very bizarre issue that was specifically dealing with Windows' OS internals. But then, a lot of what makes upgrading easy or difficult depends on the nature of the codebase in the first place - if you have code assist, typechecking, linters and whatnot set up, then upgrading is far easier, and it's just unfortunate that these things weren't so widely used in the past when a lot of now-hard-to-upgrade projects got started. It's also worth pointing out that, on the whole, the Python developers now, with hindsight, mainly agree that Python 3 went too far with its BC breaks, and there seems to be a general consensus to treat that as a disaster never to be repeated. Yes, they do remove things now and then, but it's nowhere near as bad as a lot of other things (web standards, for starters). I think Python 3 takes a pretty decent middle ground between "move fast and break things" and the opposite extreme of "never break anything ever", which itself has its own downsides.

That said, I don't really want this to turn into a Python 2 vs Python 3 argument - there are enough of those out there on the internet already, and I think most of us just want to treat the whole Python 2 to 3 transition with a "water under the bridge" attitude.

As for Perl 5, while it's not my favorite language to develop in, and I only have a limited bit of experience with it from several years ago, it certainly struck me as a good solid language that just doesn't break on you, especially after the decision was made to make Raku its own thing and make clear that Perl 5 would live on. So although I have a personal preference for Python, and would find that far easier to work with myself, I wouldn't mind if things ended up moving to Perl instead.

the Linux distros have just soured on it and bought into the finger-wagging "it's obsolete, you must upgrade" attitude on this, not having much respect for the fact that it's a totally new language

Actually, a lot of them, the vast majority, don't seem to understand that a build and dev environment isn't necessarily the same as a normal use environment. e.g. python 2.7 may be "insecure" but it will not be if it's only being used with known and trusted scripts. There's a big divide between using tools with known scripting or using it with untrusted/foreign/third-party scripting.

On this, I agree with you. There's a lot of rhetoric out there that just unconditionally insists "everything old is bad", which is silly, really. However, I also can see where the distro maintainers are coming from: first, maintaining more versions of something than necessary is additional effort that they don't want to have to expend, and second, if they did continue to supply Python 2 forever even with the express intent that it only be used for certain purposes (like building things using known scripts), then there'd be a lot of people out there who would just say "oh, it's there, clearly I can use it for anything I like then" - basically, it's a case of protecting users against themselves. So, on the whole, it's probably for the best that distros removed Python 2, while python.org still supplies the source to old versions, which can (as I've discovered!) be manually built pretty easily and then used for whatever they're truly needed for.

[Moonchild]

For comparison, I recently upgraded a decently sized project from 3.7 to 3.10

See, there might be a repeat problem here if minor point versions require changing scripts. We might end up on a specific point version of python unless we have a dedicated maintainer if each smaller release of python comes with breaking changes (PHP got bad like that as well with 7.1, 7.2, 7.3, etc.) and that inevitably also reduces the portability of UXP across distros which I'd like to avoid. So then the question becomes "when will it break next?" and "when will distros stop carrying the point version we are using now?" because it's not possible for a code base our size to constantly update the build system and plumbing for rolling releases of dev languages with breaking changes.
I wasn't aware a perl implementation was on the table either. Ultimately I have no strong preference other than what is the best to maintain going forward with our limited resources. I know no perl, myself. I know little python out of necessary exposure.

[athenian200]

FWIW, I don't really agree with your analysis.

Well, actually the thing about Python 3 being a brand new language wasn't my analysis, I was told that repeatedly by a former associate who knew more about Python than me until it was pounded into my head as a fact, and it seemed plausible, because why else wouldn't we just inspect Mozilla's bugs and move to Python 3? Could very well I've been proceeding from a faulty assumption, but I got the impression he was an expert on build systems and Python, so while I questioned a lot of the reasoning on other things, I figured something like that was well within his domain of knowledge.

That said, I don't really want this to turn into a Python 2 vs Python 3 argument - there are enough of those out there on the internet already, and I think most of us just want to treat the whole Python 2 to 3 transition with a "water under the bridge" attitude.

Didn't really mean it to be, just was trying to justify why we're not on Python 3, because the question always puts me on the defensive a bit subconsciously. It's not that I am a fan of Python 2... far from it, I actually find Python 2 counter-intuitive and really hard to work with. I'm just a C++ developer who knows more Perl than Python, and feels a bit stuck using Python 2 because a overly-complicated build system we inherited relies on it, I probably wouldn't even care that Python 3 broke things if I weren't in this situation myself, to be honest. The frustration is mostly that the deprecation of Python 2 means we constantly have to defend ourselves for using it in our codebase and are under a lot of pressure to put work into something we don't really understand that won't really move UXP itself forward in a meaningful way other than being a little easier to develop for, even though it's not a part of Pale Moon itself and doesn't present any security risks to end-users.

As for Perl 5, while it's not my favorite language to develop in, and I only have a limited bit of experience with it from several years ago, it certainly struck me as a good solid language that just doesn't break on you, especially after the decision was made to make Raku its own thing and make clear that Perl 5 would live on. So although I have a personal preference for Python, and would find that far easier to work with myself, I wouldn't mind if things ended up moving to Perl instead.

Yeah, and that would have been a long-term project to be done over the course of years, not something we get done in a few months. But if you have the Python 3 expertise, and you think that they won't pull this again with a Python 4, then I would be happy to collaborate with you on moving the build system to Python 3. I just thought we had dimissed the idea of ever moving to Python 3, and that it was off the table since we all lack the Python expertise, so I had been occasionally considering what else we could do to move away from Python 2 other than go to Python 3. The reasoning was, "Well, if we have to rewrite the whole build system for the new Python 3 language anyway, why not just rewrite the whole build system for Perl then, since I already know that and won't have to master as many new skills to rewite the code base in Perl as I would to have to rewrite in Python 3?"

[vannilla]

A long (long) time ago I took a peek at the issues revolving around the Python 2 vs 3 in the UXP build system and I found out that most of them are either in vendor packages or in the mozconfigure module.
Mach itself is relatively small and the biggest issue would be to port over the machinery to allow new commands through annotations/decorators, but even then it's mostly a syntactical issue, if I remember correctly.
Speaking of mozconfigure, apparently it contains a hand-written parser for Autoconf 2.13, which is why newer versions of Autotools cannot be used.
If I remember correctly, mozbuild also contains a hand-written implementation of the UNIX make tool; apparently at Mozilla they wanted to get rid of make entirely and rewrote the tool in Python 2. Given the fact that they have parsers, the conversion here will definitely have most of the issues between string handling, difference in operators and whatnot.

[Moonchild]

I decided to split this off into its own thread and leave the howto focused on that

[Keiji]

For comparison, I recently upgraded a decently sized project from 3.7 to 3.10

See, there might be a repeat problem here if minor point versions require changing scripts. We might end up on a specific point version of python unless we have a dedicated maintainer if each smaller release of python comes with breaking changes (PHP got bad like that as well with 7.1, 7.2, 7.3, etc.) and that inevitably also reduces the portability of UXP across distros which I'd like to avoid. So then the question becomes "when will it break next?" and "when will distros stop carrying the point version we are using now?" because it's not possible for a code base our size to constantly update the build system and plumbing for rolling releases of dev languages with breaking changes.
I wasn't aware a perl implementation was on the table either. Ultimately I have no strong preference other than what is the best to maintain going forward with our limited resources. I know no perl, myself. I know little python out of necessary exposure.

Python is now on an annual release cadence. What that means for us is that once a year, someone needs to check the Python release notes for any deprecations, follow the recommended mitigations, and test. That is not hard to do for a codebase that is correctly set up to support this workflow moving forward, and as long as I can get us migrated to Python 3 in the first place (which is the difficult bit), I'm totally happy to do that once-a-year upgrade thereafter.

A long (long) time ago I took a peek at the issues revolving around the Python 2 vs 3 in the UXP build system and I found out that most of them are either in vendor packages or in the mozconfigure module.
Mach itself is relatively small and the biggest issue would be to port over the machinery to allow new commands through annotations/decorators, but even then it's mostly a syntactical issue, if I remember correctly.
Speaking of mozconfigure, apparently it contains a hand-written parser for Autoconf 2.13, which is why newer versions of Autotools cannot be used.
If I remember correctly, mozbuild also contains a hand-written implementation of the UNIX make tool; apparently at Mozilla they wanted to get rid of make entirely and rewrote the tool in Python 2. Given the fact that they have parsers, the conversion here will definitely have most of the issues between string handling, difference in operators and whatnot.

Thanks - this is really useful info which will help out with taking stock of what needs doing.

[vannilla]

Thanks - this is really useful info which will help out with taking stock of what needs doing.

Since this is now a dedicated thread, here's a few more words of advice: most vendor packages (i.e. mozconfigure/mozbuild dependencies) are either overkill because the actually used features are so simple they can be implemented without depending on third-party code, or can be substituted with built-in Python 3 modules. This is of course not true for every single package, but when I checked it was around the 3.2 era and to be honest I wouldn't be surprised if going forward the number of useful third-party modules becomes zero.
Another thing to keep in mind is that a good bunch of mach commands are actually useless, mostly because they reference obsolete tools or services that UXP development does not depend on, so it should be safe to discard them during the porting, reducing the workload maybe even significantly if it entails getting rid of dependencies.
I think there was something else, but I can't remember right now. If I recall it I'll post it here.

[Keiji]

I had suspected that might be the case! Yes, dropping unneeded dependencies is always a good way of making things easier :)