I created a Python 3 port of UXP out of boredom...

[athenian200]

Well, also obligation because a user complained about not being able to build Epyrus on newer Linux. But mostly boredom and curiosity.

https://repo.palemoon.org/athenian200/e ... ch/python3

https://repo.palemoon.org/athenian200/U ... hon3-dirty

So, unfortunately the commit history is terribly sloppy because I wasn't really expecting to get very far at first, and didn't do the port in a systematic way, mostly just fixing issues as I hit them... it's also currently full of debug prints. It may take as long to clean up/document everything done in this port and make sure everything is good enough as it did to create it in the first place, because there's a lot of work in this that needs to be properly documented and redone cleanly.

But yeah, there are several things to note about this port:

1. It doesn't actually use virtualenv like it claims, I got around touching most of that code (which was apparently a blocker before) by using Python 3's native venv plugged into the old virtualenv machinery, effectively hijacking it and adapting it without properly renaming anything. This also means it can't use the Python 3 that's bundled with MozillaBuild (which seems to be deliberately built without venv), and requires installation of the official Python for Windows package along with manually adding it to your PATH in MozillaBuild while making sure it doesn't pick up any of the bundled Pythons. Have not decided yet if this was a good approach or I need to go back and fix virtualenv instead.

2. Originally it couldn't find application.ini.h in $topobjdir/build due to a malformed path that was being constructed in a way I didn't understand... was able to work around that by putting using a local Makefile directive that basically goes around Python path generation (though a relpath like ../../build/application.ini.h directly as a cpp include works too). It seems like the issue randomly disappeared last time I checked while I was working on other stuff, but I am not sure if it's truly fixed or it is intermittent.

3. Early on in development, I had trouble getting Python 3 to execute autoconf on Windows in the MSYS1 environment. I worked around this for a while by generating a frozen autoconf.m4f file, which got me further in the process... at some point it seems like it mysteriously wasn't needed, and again I have no clue why or if other people's systems might suddenly need it. But my difficulties interfacing between MSYS1 on Windows and Python 3 early on led to me creating a custom check_cmd_output function that sits between the terminal and a raw subprocess.Popen call, basically trying to clean up the output enough that things work, because the default output from Python 3 on Windows MSYS1 is... not pretty, it makes all kinds of wrong assumptions that have to be compensated for.

4. On Linux, libfreebl3priv.so is not generated at all and you get libfreebl3.so instead, which will cause an install-manifest error when packaging most UXP applications. I'm not really sure what the difference between the two is (they're nearly binary identical except for one byte), but for some reason only one is generated, and it's not the one normally packaged. Normally both get generated. I am a bit concerned about whether that's an indicator of deeper problems that would have required a better approach to start with, but it's possible to work around it... if needed by renaming libfreebl3.so and editing the ELF header or something before packaging.

5. Symlinks in $topobjdir/dist/bin are busted on all platforms, only NSDISTMODE=copy or LD_LIBRARY_PATH pointed at $topobjdir/dist/bin allows mach run to work, just like on my SunOS port. Whatever magic was holding that together on Linux broke with the Python 3 port, and honestly I never understood how that aspect of the build system works. On Windows, Python 3 apparently figured out how to generate Windows symlinks when it was falling back to copy mode before, which is what caused issues. On Linux, it's really not clear why Python 2 has mach run working properly on Linux without an $ORIGIN flag, LD_LIBRARY_PATH, or NSDISTMODE=copy, but Python 3 breaks this and suddenly requires a manual workaround like what is required for BSD or SunOS.

6. Python 3 compatibility required a lot of work on XPIDL generation, IPDL generation, and WebIDL generation, with later Python 3 versions even requiring the modification of regexes that became invalid, messing around with the AST, and tweaking the lexer to avoid new recursion limits. While I got the port to a point where IPDL headers are included exactly enough times to compile (not too many, not too few), and XPIDL/WebIDL seems to generate correctly enough that the build doesn't fail and the application launches, I'm not at all confident that everything is 100% correct (given all the strange nits I described) and that there wouldn't be any subtle issues that would only show up in a long browsing session.

7. The last few versions of Python removed LooseVersion and StrictVersion, as well as platform.dist which allowed you to determine your Linux distro. There were no replacements, and the Python devs basically told everyone to have fun hunting for random third-party libs to replace built-in functionality. Worked around that by directly working with tuples of ints in an ugly way to fill in the version logic, since that's mostly what it was doing anyway. As for platform.dist, I filled in that gap by just reading /etc/os-release directly. Easier than the rest of the port, but definitely is another decision I'm not confident about and may need to revisit in favor of a third-party package or two later on.

8. For some reason the build proceeds without ALSA development libraries on the Python 3 version, but stops you from proceeding on the Python 2 version... another discrepency.

9. There are likely still several dusty corners of the build system with unmodified Python 2 code that could be mechanically replaced or fixed easily, only enough to get things building, running, and packaging was really done. It has not been tested on anything but Fedora Linux and Windows 10.

10. The EnumStrings that test whether something is OS_TARGET or OS_ARCH are enforced strictly, and when Linux attempts to set one equal to the other, OS_TARGET gets overwritten with OS_ARCH. My temporary hack was to simply switch most instances of OS_TARGET to OS_ARCH, but it could likely also be solved by figuring out how to fix the list append logic so it acts like Python 2 and takes an enum from either list, adding the missing enums so the lists are equal to begin with, or possibly just discarding OS_TARGET and just using OS_ARCH for everythong... no idea which approach is best.

11. I stubbed out a lot of the old processhandler logic, it was very brittle on Windows in particular and required hooking into the exact function signature of something in Python that can change from version to version, which sounded like a bit of a difficult hack to maintain... and honestly it seemed like whatever it was doing wasn't compatible with Python 3. Not sure how big a difference it makes, but it could make builds a bit slower, I think.

I was really debating with myself whether to share this at all in its current state, but I decided if I wait until it's perfect I might be working at it for a very long time. If you want to mess around with it, you're free to do so... just be prepared for it to have a lot more rough edges than our reliable Python 2 build system.

[Basilisk-Dev]

Semi-related since you mentioned a user asking about Python 3, but last week I was thinking about the Python 2 issue on newer Linux distros. I considered making some static binaries of Python 2 for Linux x86_64 and aarch64 so that when users complain that their OS/distro doesn't have Python 2 we could just point them there and say to use those binaries.

Also this is cool, happy to see work happening towards Python 3 support. It might be worth reviewing SeaMonkey's implementation since they ported their build system to Python 3 a few years ago and they were close enough to our fork point that it should be similar-ish.

[vannilla]

Semi-related, but with Python 3 slowly creeping in I wonder if Autoconf 2.13 can also be replaced somehow.
I know how to use Autoconf and can migrate it over newer versions or even remove it entirely, but I can't figure out how it's used by the build system at all.

[Moonchild]

I can't figure out how it's used by the build system at all.

It's used in the build system for both configure and for the plethora of make files.
The build system is, tbh, a mess we inherited from Mozilla, and it's just not easy to change it.

[athenian200]

Semi-related, but with Python 3 slowly creeping in I wonder if Autoconf 2.13 can also be replaced somehow.
I know how to use Autoconf and can migrate it over newer versions or even remove it entirely, but I can't figure out how it's used by the build system at all.

Oh, actually our codebase apparently vendored Autoconf 2.13 a long time ago, so we only need m4 to build now. Autoconf back then was little more than a series of m4 macros. So we actually don't need to get rid of it, and I think MC told me once he wanted to keep it.

The main reason for keeping it is so that UXP applications can still rely mostly on Makefiles and shell scripts without having to worry about Python too much, if they prefer. The setup we have basically confines Python to UXP and lets applications pass in shell, mozconfig, and Make directives, with maybe a few Python stubs. I believe it is also responsible for version checks in old-configure, and basically turning old-configure.in into an old-configure file.

But yeah, if we're talking theoretical we could remove Autoconf if we wanted, it's basically just macros. Python already does the heavy lifting in the build system, and Mozilla did it upstream eventually. In all honesty, I get the temptation, my Python 3 port actually would have been easier if we didn't have to keep Autoconf... getting Python 3 to interface with Autoconf (particularly on Windows MSYS1) was almost more of a pain than updating the code more generally, and nearly stopped the project before it got started. The Python 3 bundled with MozillaBuild worked so poorly that I wound up having better luck with a standard Windows build of Python 3 forced into MSYS1 via PATH manipulation, which was a desperation move made early on that I kept building on.

https://repo.palemoon.org/athenian200/U ... heckcmd.py

I wound up replacing several raw subprocess.Popen calls with this wrapper just because of how much I was having to normalize newlines and clean up the mismatched assumptions. Python 2 was assuming ASCII, MSYS1 was assuming ASCII, and a lot of encoding issues were avoided originally by treating everything as raw bytes... but Python 3 is a lot more strict about treating text as text and bytes as bytes, and thus keeps translating between UTF-8 and CP1252, and sometimes I have to normalize newlines from Windows to Unix format when dealing with terminal output because Python knows it is running on Windows and tries to use DOS-style newlines, but MSYS1 expects Unix style line endings I think, so all kinds of guarding against wrong encodings being passed around and erroring out is now needed. Still, the task wasn't insurmountable, programmers have been dealing with these old DOS/Unix newline issues and Unicode/ASCII issues for decades...

[mittorn]

The build system is, tbh, a mess we inherited from Mozilla, and it's just not easy to change it.

I think, it should be possible to implement mozbuild support for waf buildsystem:
https://waf.io/
It supports both python2 and python3, have project generators for ide and powerful task management.
It does not use make, but implements task running and dependency tracking in python, with checksum checking, so it might be little slower than mozbuild

[athenian200]

https://repo.palemoon.org/athenian200/U ... hon3-clean

So... now it's cleaned up a bit. Obviously the branch is very out of date given how much time was spent, so it still couldn't be applied as-is, but thanks to the cleaner commit history, it should be easier to rebase this on master and clean up any conflicts as they arise.

So, this actually addresses most of the problems with the original port...

freebl3priv.so now seems to have the right name on Linux. application.ini.h is now found properly, and the dist/bin symlinks appear to work on Linux again. And I found a way to get it to use either kernel or OS enums for OS_ARCH and OS_TARGET (like we currently do) rather than restricting us to one list or the other for mozbuild files. I was also able to actually fix mozprocess rather than just stubbing it out.

The one thing I couldn't actually save in the end was virtualenv 15.x. I got it to work on Python up to 3.10 just to have it as a reference comparison against venv to make sure there weren't behavior differences (was paranoid after the dist/bin symlinks stopped working on Linux with my first build). There were none... apparently this build system is not dependent on virtualenv 15.x quirks at all... it would even run under system Python if you copied all the .pth files it generates into your system Python's site-packages and made them into absolute paths. virtualenv was apparently only ever a convenient way to make sure that developers didn't have their system Python polluted by references to mach, and that mach didn't accidentally find versions of packages on a developer's system that weren't what was shipped with the browser.

But yeah, this should work almost exactly like our current build system with minimal behavior changes, other than mozbuild being way more sensitive to incorrect enums and blowing up really early on in the build rather than when it hits the offending mozbuild file.

There is one BIG issue I just thought of as I was finishing this work, though... if a developer needs to go back in time through git history to before we did the Python 3 switch... they would still need Python 2.x in order to build the system at that point. I've been thinking about whether this needs to be addressed... and I came up with two basic ideas...

The first is that, after we've made sure this new version of our build system is stable (I honestly think it should probably be tested for months before we even think about making it the standard), it touched every Python file in our tree), we might have to go back in our git history to the very first commit and swap out the build system there, and then replay every later commit on top of that new version. That would probably be better to do on a new repo (or at least a new branch), effectively archiving the Python 2 one as a backup just in case we need to see the real history again.

The second potential idea I had, is that alongside something like that, it might make sense to gather up all the scattered Python scripts across our tree and unify them into a location like build or python, adapting them to work there, so that we can split platform/build and platform/python into submodules, and thus people would be able to move the build system and its Python stuff forward or backwards in time independently of the actual C/C++/JavaScript source code we usually work on. This might sound like overengineering, but I really think this could be a good application for the git submodule idea we're already using for the application directory. It would also mean if someone does want to build the Python 2 version of the build system, they could do that.

None of that really has to be done right away, though... and really the biggest obstacle to a Python 3 migration is simply the need to test it enough given how extensive the changes are. The problem is, we can't do that migration piecemeal like Mozilla did... Python 2 is 100% dead at this point, so every part of the tree has to be migrated in one go, or nothing will work on systems that lack Python 2. They migrated during a grace period where the harder stuff could be done later while the build system just mostly worked on Python 3... we won't get that luxury and basically have to do this all at once or not at all.

[Moonchild]

There is one BIG issue I just thought of as I was finishing this work, though... if a developer needs to go back in time through git history to before we did the Python 3 switch... they would still need Python 2.x in order to build the system at that point. I've been thinking about whether this needs to be addressed... and I came up with two basic ideas...

As already touched on on the repo, I don't think this is a big issue. We've had multiple breaking changes in history. e.g. if you go back far enough, then Pale Moon will stop building on MSVC because VS2022 isn't backwards compatible to the specifics of very old code bases that were developed against, say, 2008 or older compilers.
On Windows it won't even be much of an issue since we do have python 2 and 3 in our mozilla build tools, just needing to be references properly.
It doesn't matter when you make this changeover to python3, delaying it because of perceived build issues won't be a problem any more now than it will in the future; the breaking changes won't go away by waiting until it becomes impossible to build on more systems...

[mittorn]

Why even buildsystyem depend on virtualenv? Do we need this complexity?
I've tried to build on x32-abi system few years ago and it refused to work

[athenian200]

Apparently the port is in a better state than I thought...

It seems like it compiles on platforms I hadn't even tested it with, and most of what remained to be fixed involved stuff I had avoided by testing in a controlled environment, essentially ensuring Python 2 wouldn't get pulled in, whereas on real UXP developer systems, Python 2 has a high chance of being available in the PATH and getting pulled in by mistake.

[Drugwash]

Is there a way for the user to select the specific Python version to be used in the virtual environment, in mozconfig (preferrably) or wherever?

I mean, on [older] systems where the user is for some reason forced to self-build, there can be multiple versions of Python3 installed, and at some point the official sources might require a higher version than what is default for the given system; i.e. on Ubuntu Bionic and derivatives the default is Python 3.6 but there can also be any of Python 3.7, 3.8 and higher backports (up to 3.12 currently), so assuming your port would at some point require a minimum of Python 3.9 how could the user specify the Python 3.9 executable for the UXP/Pale Moon Python virtual environment?

Or would an initial script purposely scan the system for the required minimal Python3 version, and inform the user if not found?

Maybe I'm overthinking this but I believe it would be beneficial to future-proof the basis at least for as long as the UXP (and Pale Moon) sources will still compile on older systems.

[athenian200]

Is there a way for the user to select the specific Python version to be used in the virtual environment, in mozconfig (preferrably) or wherever?

I mean, on [older] systems where the user is for some reason forced to self-build, there can be multiple versions of Python3 installed, and at some point the official sources might require a higher version than what is default for the given system; i.e. on Ubuntu Bionic and derivatives the default is Python 3.6 but there can also be any of Python 3.7, 3.8 and higher backports (up to 3.12 currently), so assuming your port would at some point require a minimum of Python 3.9 how could the user specify the Python 3.9 executable for the UXP/Pale Moon Python virtual environment?

Or would an initial script purposely scan the system for the required minimal Python3 version, and inform the user if not found?

Maybe I'm overthinking this but I believe it would be beneficial to future-proof the basis at least for as long as the UXP (and Pale Moon) sources will still compile on older systems.

Actually, I tested against a variety of versions... and I will share with you the trick I used during development. Essentially, you can just create a directory somewhere on your system with symlinks to the version of Python you want. Let's say /usr/local/bin just for example (which is what I used). In that directory, create symlinks to python3 and python that both point to your preferred Python version. Then, inside the terminal window you're using to build Pale Moon, modify PATH with something like "export PATH = /usr/local/bin:$PATH" and then you should have your preferred version of Python be the default available, since that directory will now overshadow system Python. To revert to the standard, just set PATH back to normal. If you're testing multiple Python versions that are altinstalled to the same location like I was, you can even just keep creating and deleting those symlinks and get the environment to point to different Python versions without many further changes.

So I don't necessarily think anything needs to be implemented at the build system level to achieve this.

Just so you know (and maybe feel a bit reassured), I implemented this in such a way that it should support all the way back to Python 3.4... it was mostly a result of starting off with older versions because it was easier to get older Python 3 working there and walk up through the versions because the deprecations hit in big waves that way rather than all at once (especially since I originally was trying to keep virtualenv). And also, MozillaBuild on Windows is still using Python 3.7 something, and as far as I can tell there's no real advantage to using a newer Python 3 version... I just want to support the newer versions primarily for mainstream Linux distros, but there's no reason to force people to use the latest. Everything we'll ever need from Python 3.x is in 3.4+, and my plan is to try and support newer versions in a way that doesn't break older ones if I can. Continuing to support Python 2.x would be excessive, IMO, but older Python 3 is more than doable.

It is worth remembering that this was a Python 2.x codebase originally, and that basically I only did what was needed to get it working on newer Python... there was not exactly an eager desire to take advantage of new Python features, we just wanted people on modern Linux to be able to build our stuff like they did before finger-wagging distro maintainers and Guido von Rossum's people put their collective feet down and said, "No, you can't use Python 2 anymore even if it's just for a big set of glorified Makefiles, you are going to put in two months of hard work to support our Python 3 vision, or it's either forcing people to hunt down ancient OpenSSL versions to build Python 2 from scratch, hoping people know what Tauthon is, or full-blown Docker containers for your project, and we don't care how that affects you."

[Drugwash]

If you're testing multiple Python versions [...]

Personally I haven't gotten to the point that I'd have to thoroughly test multiple versions. For my own needs (using Linux Mint 19.2 Cinnamon x64) the default Python 3.6 usually is enough, all my scripts use it. However I do ocasionally stumble into certain additional modules that would make life easier, and which unfortunately require higher Python versions. Using basic/default (internal) Python modules for certain tasks may be tedious. In some cases I do attempt to test some script in higher Python versions (I do have minimal 3.7, 3.8, and 3.12 installed along with default 3.6) just to see how it would behave in other (newer) systems, but I simply call the particular version I want through script's shebang (i.e #!/usr/bin/env python3.12); dunno how that would work in Windows or MacOS though . For some reason I never fancied the idea of a virtual environment (which is why I also stay away from AppImage and Flatpak). That's why I find it kinda difficult to set up, and would like the easiest way possible to set up UXP/Pale Moon (and/or whatever else using the UXP platform). Using additional symlinks might [theoretically] prove problematic for other applications if they scan those locations for Python executables.

But if you say an old[er] version of Python3 would be more than enough for the long run then I (and other users of older systems) shouldn't be worried. After all Python2 has been perfectly suitable so far, and only its recent demise from newer systems (due to its official deprecation) makes this switch to Python3 necessary. I do realize though that newer Python3 versions have become increasingly difficult to use, especially due to the mandatory use of virtual environments instead of allowing the installation of additional modules or updating original ones system-wide. Hopefully it won't be very challenging to get these newer versions to work while keeping compatibility with older versions, in UXP.

Keep up the good work!

[jobbautista9]

finger-wagging distro maintainers and Guido von Rossum's people put their collective feet down and said, "No, you can't use Python 2 anymore even if it's just for a big set of glorified Makefiles, you are going to put in two months of hard work to support our Python 3 vision, or it's either forcing people to hunt down ancient OpenSSL versions to build Python 2 from scratch, hoping people know what Tauthon is, or full-blown Docker containers for your project, and we don't care how that affects you."

Off-topic:
And to that behavior I'd personally reply "okay, don't expect for any more mainstream adoption of your GNU/Linuxes, BSDs and whatever unix crap you have then, because here in my Windows 11 I can just easily download an old build environment packaged up into a convenient installer. Come back when you have figured out what universal packaging solution you've all adopted instead of fragmenting yourselves and calling that a "feature". "

[athenian200]

Well, it looks like my changes were merged in. Just in time for my birthday, incidentally. Though just to reassure anyone still wondering about our support for older systems, I'd like to show one post from MC on the PR thread:

So what do we do? Our buildtools has 3.7.9; Oracle Linux which is being used for our Linux builds has 3.6.8.
We probably need to remain conservative, looking at that. (emphasis mine)

So that effectively means that on top of me testing with older Python 3 versions and finding them to feel more "solid" than Python 3.11+, it's worth keeping in mind that MozillaBuild on Windows only integrates 3.7.9, and our preferred Linux build environment integrates 3.6.8, so that means it's unlikely for purely pragmatic reasons we'll require anything newer than 3.6. And I tested even further back than 3.6 while working on this. In several places I set this up to try newer Python 3 features and fallback to older behaviors in order to ensure compatibility on 3.11+ and versions as low as 3.4, though that was probably overkill and was mostly done to drag Virtualenv up through several Python versions before ultimately ditching it when Python 3.11+ made the old virtualenv behavior incompatible, essentially making the virtualenv vs. venv decision for us.

In any case, hope that this version of the build system proves stable for everyone. I tested it in every way I could think of. And I'm standing by ready to try and fix issues that come up.

[__Sandra__]

I'm trying to compile the current branch and I'm getting an error. I'm compiling on Windows 7 with VS2019 and MozBuild 3.4. This configuration worked fine until now. Now, after the log entry "Creating Python environment", a Windows error message appears "powershell.exe not found."

[Drugwash]

Well, it looks like my changes were merged in. Just in time for my birthday, incidentally.

Congratulations, and Happy Birthday!

[athenian200]

I'm trying to compile the current branch and I'm getting an error. I'm compiling on Windows 7 with VS2019 and MozBuild 3.4. This configuration worked fine until now. Now, after the log entry "Creating Python environment", a Windows error message appears "powershell.exe not found."

That is odd, I didn't explicitly add a Powershell requirement while working on this, so I wouldn't expect that to be a problem... I'll look into it for sure. I admittedly did not test Windows 7 as a build environment, though.

[athenian200]

https://repo.palemoon.org/athenian200/U ... 774aa2e9c5

https://repo.palemoon.org/athenian200/U ... on3-clean/

Can you try this branch really quick? I probably want to try and make a new branch for this rather than use my old one, but for now, go ahead and try this one...

This kind of issue got me so frustrated during my "dirty" build that I wound up changing almost every callsite in the tree that uses subprocess.Popen with a helper called "check_cmd_output" that fixes everything up. But I didn't want to have to commit that (because of uncertainty on how it would impact other platforms), and I was sure I'd resolved the issue more cleanly.

I guess I should have tested more Windows versions... sorry guys. Anyway, I didn't have my schedule clear today to work on this past noon, so I might not be able to more thoroughly test Windows 7 and 8 until later tonight. I kind of got blindsided by this and was hoping that a major issue like this would be caught in testing.

[__Sandra__]

https://repo.palemoon.org/athenian200/U ... 774aa2e9c5

This patch solved the problem of stopping the build, but the "powershell.exe not find" message still appears.

There is also a problem with the display in the console. There used to be Russian letters here.

codepage.png

My build is about 150 minutes long, I won't have a chance to tell you how it turns out. But judging by the beginning of the log everything will be fine.