I created a Python 3 port of UXP out of boredom...

[athenian200]

Well, also obligation because a user complained about not being able to build Epyrus on newer Linux. But mostly boredom and curiosity.

https://repo.palemoon.org/athenian200/e ... ch/python3

https://repo.palemoon.org/athenian200/U ... hon3-dirty

So, unfortunately the commit history is terribly sloppy because I wasn't really expecting to get very far at first, and didn't do the port in a systematic way, mostly just fixing issues as I hit them... it's also currently full of debug prints. It may take as long to clean up/document everything done in this port and make sure everything is good enough as it did to create it in the first place, because there's a lot of work in this that needs to be properly documented and redone cleanly.

But yeah, there are several things to note about this port:

1. It doesn't actually use virtualenv like it claims, I got around touching most of that code (which was apparently a blocker before) by using Python 3's native venv plugged into the old virtualenv machinery, effectively hijacking it and adapting it without properly renaming anything. This also means it can't use the Python 3 that's bundled with MozillaBuild (which seems to be deliberately built without venv), and requires installation of the official Python for Windows package along with manually adding it to your PATH in MozillaBuild while making sure it doesn't pick up any of the bundled Pythons. Have not decided yet if this was a good approach or I need to go back and fix virtualenv instead.

2. Originally it couldn't find application.ini.h in $topobjdir/build due to a malformed path that was being constructed in a way I didn't understand... was able to work around that by putting using a local Makefile directive that basically goes around Python path generation (though a relpath like ../../build/application.ini.h directly as a cpp include works too). It seems like the issue randomly disappeared last time I checked while I was working on other stuff, but I am not sure if it's truly fixed or it is intermittent.

3. Early on in development, I had trouble getting Python 3 to execute autoconf on Windows in the MSYS1 environment. I worked around this for a while by generating a frozen autoconf.m4f file, which got me further in the process... at some point it seems like it mysteriously wasn't needed, and again I have no clue why or if other people's systems might suddenly need it. But my difficulties interfacing between MSYS1 on Windows and Python 3 early on led to me creating a custom check_cmd_output function that sits between the terminal and a raw subprocess.Popen call, basically trying to clean up the output enough that things work, because the default output from Python 3 on Windows MSYS1 is... not pretty, it makes all kinds of wrong assumptions that have to be compensated for.

4. On Linux, libfreebl3priv.so is not generated at all and you get libfreebl3.so instead, which will cause an install-manifest error when packaging most UXP applications. I'm not really sure what the difference between the two is (they're nearly binary identical except for one byte), but for some reason only one is generated, and it's not the one normally packaged. Normally both get generated. I am a bit concerned about whether that's an indicator of deeper problems that would have required a better approach to start with, but it's possible to work around it... if needed by renaming libfreebl3.so and editing the ELF header or something before packaging.

5. Symlinks in $topobjdir/dist/bin are busted on all platforms, only NSDISTMODE=copy or LD_LIBRARY_PATH pointed at $topobjdir/dist/bin allows mach run to work, just like on my SunOS port. Whatever magic was holding that together on Linux broke with the Python 3 port, and honestly I never understood how that aspect of the build system works. On Windows, Python 3 apparently figured out how to generate Windows symlinks when it was falling back to copy mode before, which is what caused issues. On Linux, it's really not clear why Python 2 has mach run working properly on Linux without an $ORIGIN flag, LD_LIBRARY_PATH, or NSDISTMODE=copy, but Python 3 breaks this and suddenly requires a manual workaround like what is required for BSD or SunOS.

6. Python 3 compatibility required a lot of work on XPIDL generation, IPDL generation, and WebIDL generation, with later Python 3 versions even requiring the modification of regexes that became invalid, messing around with the AST, and tweaking the lexer to avoid new recursion limits. While I got the port to a point where IPDL headers are included exactly enough times to compile (not too many, not too few), and XPIDL/WebIDL seems to generate correctly enough that the build doesn't fail and the application launches, I'm not at all confident that everything is 100% correct (given all the strange nits I described) and that there wouldn't be any subtle issues that would only show up in a long browsing session.

7. The last few versions of Python removed LooseVersion and StrictVersion, as well as platform.dist which allowed you to determine your Linux distro. There were no replacements, and the Python devs basically told everyone to have fun hunting for random third-party libs to replace built-in functionality. Worked around that by directly working with tuples of ints in an ugly way to fill in the version logic, since that's mostly what it was doing anyway. As for platform.dist, I filled in that gap by just reading /etc/os-release directly. Easier than the rest of the port, but definitely is another decision I'm not confident about and may need to revisit in favor of a third-party package or two later on.

8. For some reason the build proceeds without ALSA development libraries on the Python 3 version, but stops you from proceeding on the Python 2 version... another discrepency.

9. There are likely still several dusty corners of the build system with unmodified Python 2 code that could be mechanically replaced or fixed easily, only enough to get things building, running, and packaging was really done. It has not been tested on anything but Fedora Linux and Windows 10.

10. The EnumStrings that test whether something is OS_TARGET or OS_ARCH are enforced strictly, and when Linux attempts to set one equal to the other, OS_TARGET gets overwritten with OS_ARCH. My temporary hack was to simply switch most instances of OS_TARGET to OS_ARCH, but it could likely also be solved by figuring out how to fix the list append logic so it acts like Python 2 and takes an enum from either list, adding the missing enums so the lists are equal to begin with, or possibly just discarding OS_TARGET and just using OS_ARCH for everythong... no idea which approach is best.

11. I stubbed out a lot of the old processhandler logic, it was very brittle on Windows in particular and required hooking into the exact function signature of something in Python that can change from version to version, which sounded like a bit of a difficult hack to maintain... and honestly it seemed like whatever it was doing wasn't compatible with Python 3. Not sure how big a difference it makes, but it could make builds a bit slower, I think.

I was really debating with myself whether to share this at all in its current state, but I decided if I wait until it's perfect I might be working at it for a very long time. If you want to mess around with it, you're free to do so... just be prepared for it to have a lot more rough edges than our reliable Python 2 build system.

[Basilisk-Dev]

Semi-related since you mentioned a user asking about Python 3, but last week I was thinking about the Python 2 issue on newer Linux distros. I considered making some static binaries of Python 2 for Linux x86_64 and aarch64 so that when users complain that their OS/distro doesn't have Python 2 we could just point them there and say to use those binaries.

Also this is cool, happy to see work happening towards Python 3 support. It might be worth reviewing SeaMonkey's implementation since they ported their build system to Python 3 a few years ago and they were close enough to our fork point that it should be similar-ish.

[vannilla]

Semi-related, but with Python 3 slowly creeping in I wonder if Autoconf 2.13 can also be replaced somehow.
I know how to use Autoconf and can migrate it over newer versions or even remove it entirely, but I can't figure out how it's used by the build system at all.

[Moonchild]

I can't figure out how it's used by the build system at all.

It's used in the build system for both configure and for the plethora of make files.
The build system is, tbh, a mess we inherited from Mozilla, and it's just not easy to change it.

[athenian200]

Semi-related, but with Python 3 slowly creeping in I wonder if Autoconf 2.13 can also be replaced somehow.
I know how to use Autoconf and can migrate it over newer versions or even remove it entirely, but I can't figure out how it's used by the build system at all.

Oh, actually our codebase apparently vendored Autoconf 2.13 a long time ago, so we only need m4 to build now. Autoconf back then was little more than a series of m4 macros. So we actually don't need to get rid of it, and I think MC told me once he wanted to keep it.

The main reason for keeping it is so that UXP applications can still rely mostly on Makefiles and shell scripts without having to worry about Python too much, if they prefer. The setup we have basically confines Python to UXP and lets applications pass in shell, mozconfig, and Make directives, with maybe a few Python stubs. I believe it is also responsible for version checks in old-configure, and basically turning old-configure.in into an old-configure file.

But yeah, if we're talking theoretical we could remove Autoconf if we wanted, it's basically just macros. Python already does the heavy lifting in the build system, and Mozilla did it upstream eventually. In all honesty, I get the temptation, my Python 3 port actually would have been easier if we didn't have to keep Autoconf... getting Python 3 to interface with Autoconf (particularly on Windows MSYS1) was almost more of a pain than updating the code more generally, and nearly stopped the project before it got started. The Python 3 bundled with MozillaBuild worked so poorly that I wound up having better luck with a standard Windows build of Python 3 forced into MSYS1 via PATH manipulation, which was a desperation move made early on that I kept building on.

https://repo.palemoon.org/athenian200/U ... heckcmd.py

I wound up replacing several raw subprocess.Popen calls with this wrapper just because of how much I was having to normalize newlines and clean up the mismatched assumptions. Python 2 was assuming ASCII, MSYS1 was assuming ASCII, and a lot of encoding issues were avoided originally by treating everything as raw bytes... but Python 3 is a lot more strict about treating text as text and bytes as bytes, and thus keeps translating between UTF-8 and CP1252, and sometimes I have to normalize newlines from Windows to Unix format when dealing with terminal output because Python knows it is running on Windows and tries to use DOS-style newlines, but MSYS1 expects Unix style line endings I think, so all kinds of guarding against wrong encodings being passed around and erroring out is now needed. Still, the task wasn't insurmountable, programmers have been dealing with these old DOS/Unix newline issues and Unicode/ASCII issues for decades...

[mittorn]

The build system is, tbh, a mess we inherited from Mozilla, and it's just not easy to change it.

I think, it should be possible to implement mozbuild support for waf buildsystem:
https://waf.io/
It supports both python2 and python3, have project generators for ide and powerful task management.
It does not use make, but implements task running and dependency tracking in python, with checksum checking, so it might be little slower than mozbuild