Autoconf dependency removal?

[ownedbywuigi]

I was looking around UXP (mainly Dactyloidae source) and saw this comment in configure.in:

Code: Select all

# Welcome to the new world of configure. We're working on moving to a
# python based configure. In the meanwhile, remains of the old autoconf
# based configure is in old-configure.in.
# If you need to add something, please come talk to the build system
# peers for now.

If this is true, can we backport whatever implementation Mozilla was cooking up with to UXP?
autoconf is the worst dependency I (and probably many others) have to deal with when building UXP, and it would be GREAT if that was removed.

[athenian200]

We actually already vendored autoconf, so this isn't really an issue anymore. No one has to have it installed on their system now. That old autoconf version is really just a series of GNU m4 scripts, nothing that scary for anyone with Unix experience...

That said, it would be kind of difficult to remove the last pieces of autoconf without getting rid of GNU make, and Mozilla actually didn't get rid of autoconf completely until fairly recently. Took them years and I think they had to use something called ninja we have no experience with to complete the transition...

Also, last time it was discussed, I don't think most of us were big fans of moz.configure, or making more stuff dependent on it... if we were to ditch autoconf, it would probably be for something different than what Mozilla came up with. Like on Windows, I'm actually pretty sure we could replace most of what autoconf and make do with MSBuild and Visual Studio tools, and it would probably work better. Though unfortunately that approach wouldn't be cross-platform.

[Moonchild]

Mozilla's later implementation is also incompatible with how we do things. If we wanted to replace our vendored stuff (which works) with another tool, we'd just be locking ourselves to another tool -- and if it's not in-tree, we'd even run the increased risk of being targeted by a supply chain attack on our build system. I'd rather not.

Also, Athenian is right: I'd rather dial back moz.configure to use our configure/make system more than moving more towards python for it. We keep running into oversights in the python parts of the build system because things are so spread out in the tree (it's a hot mess, to be quite frank). But all of those ideas will cost a lot of time to parse and work through, which is in short supply.

[athenian200]

My opinion after working with the build system a lot is basically this. Autoconf and make themselves are fine, they are fairly old programs that don't change behavior much, known quantities. However, there is one wrinkle to working with them that I am not a fan of, and that's when we start having to deal with cross-platform shell issues. Essentially, a lot of our shell scripts are written as "best effort" POSIX-style sh, which often means it only works with bash, or only bash and dash, or works in say, bash and ksh but not in dash (that actually happened once). Then there's also ash on NetBSD and similar. The problem is, shell scripts tend to rely on /bin/sh, which could be just about anything. In some ways, dealing with all these cross-platform shell issues is actually worse than Python or Perl because of the inconsistent behavior.

So my feeling is really that I would prefer to stick to Perl for anything that would otherwise be a complicated shell script with a lot of potentially platform-specific quirks. It works well with GNU Make and Autoconf as a sort of alternative to a shell invokation, but it isn't as "heavy" as Python and sits comfortably between being what Python is, and being an advanced way of writing a shell script. I guess that means my ideal preference in the end would be... Python mostly for mach and mozbuild, Perl for anything beyond the most basic shell scripts, and make/autoconf for configure. I know autoconf and make seem like unfriendly programs to newcomers, but I think after you work with moz.configure a bit and see how restrictive the sandbox is, you'll come to appreciate that the alternative is not better, and may even be more painful and complicated because of how it forces you to deal with anything beyond pointing at a directory and telling it where some sources are as if you are doing it for the whole build system rather than just the task at hand, and it also forces you to deal with the rules of yet another sandbox while doing that.

The one thing I can tell you for sure after going through the tree is that there are a lot of things done in Python in our tree that really, really should not be done in Python. We're talking codegen, writing stuff that's basically stopping just short of being a compiler, etc. In Mozilla's defense, early Python 2.x era was a much lighter program that wasn't changing rapidly. But in Python 3, a lot of this heavy lifting being done in Python rather than Perl or C is hurting us a bit on build times because Python instances are more expensive and it loads slower. Not only that, but the code is hard to maintain because Python isn't really designed for writing compilers or codegen and that stuff bitrots fast... stuff that probably should have been done in C, or using a helper like bison or flex, etc. Just... not Python.

[Basilisk-Dev]

However, there is one wrinkle to working with them that I am not a fan of, and that's when we start having to deal with cross-platform shell issues. Essentially, a lot of our shell scripts are written as "best effort" POSIX-style sh, which often means it only works with bash, or only bash and dash, or works in say, bash and ksh but not in dash (that actually happened once).

There is a Perl script online called checkbashisms that I have been using for years to avoid this exact issue. I usually install it via the package manager in whatever repo I am using, but I found a copy of it online.

https://github.com/pld-linux/checkbashi ... ashisms.pl

[Bilbo47]

My opinion [use Perl instead of the shell]

Huzzah YES variance of code interpretation across platforms is exactly what we *don't* want in a build system. While Perl can be a bear, it's at least a bear that stays consistent from one machine / OS / version to another.

[Moonchild]

Sorry to be blunt, but isn't using a known tool available across oses more consistent than generic languages like perl or Python that can and will vary version to version and are often locked to distro specifics?
So, wouldn't that promote the use of autoconf or a known shell? I mean... Bash will act the same on linux, bsd and windows... But python might not, (see e.g breaking changes between python 3.10 and 3.14... And that is even just a minor version difference...

Just hostly wondering if we're not making things harder, not easier, even regardless of the fact that I don't know Perl at all

[athenian200]

Sorry to be blunt, but isn't using a known tool available across oses more consistent than generic languages like perl or Python that can and will vary version to version and are often locked to distro specifics?
So, wouldn't that promote the use of autoconf or a known shell? I mean... Bash will act the same on linux, bsd and windows... But python might not, (see e.g breaking changes between python 3.10 and 3.14... And that is even just a minor version difference...

Just hostly wondering if we're not making things harder, not easier, even regardless of the fact that I don't know Perl at all

Well, honestly, there's no silver bullet, and that's one reason why I think there's a good argument in favor of leaving things as they are and dealing with issues as they come up, only considering migrating something off of whatever it's on if we have repeated issues in that area that would be solved by doing so (like with Python and people constantly not being able to compile Python 2 on Linux). In many cases, any change is potentially just trading one set of problems for another.

Like, if I compare shell to Python... I see it as something like two different kinds of poison. Python is mostly an instance of what I like to call, "version hell"... it changes a lot from version to version, but is fairly consistent across platforms. If I write something in Python, then for the most part I know the same version of Python will work similarly across Windows, Linux, and other platforms. Especially inside venv or virtualenv, which is part of the reason why they set that up in the first place, to avoid any distro-specific packages polluting the vendored Python packages. So Python is actually much more consistent across platforms than shell, but less consistent across versions. I know sometimes it seems like issues don't show up in MozillaBuild, but that's because it uses a locked version of Python. If you install a newer Python like Linux users have and then use PATH manipulation to make sure MozillaBuild uses it, you will see pretty much all the issues they are reporting without switching platforms (unless we explicitly gated the place where they hit the error behind a platform flag).

In contrast, shell is the reverse. Shells haven't changed much over time and have known quirks, but it's "cross-platform hell," because all the POSIX-ish utilities it relies on work a bit differently from platform to platform. Even if you wanted to standardize on bash (we currently don't, and turning it into a build requirement would likely annoy all the platforms that don't ship bash as /bin/sh), then the other utilities like grep, awk, etc... all have platform specific quirks. We would have to not only mandate use of Bash, but also GNU coreutils to make behavior consistent, which some distros like Ubuntu are moving away from even within Linux. And we'd likely have to use TOOLCHAIN_PREFIX in a lot more places and tell people to invoke the GNU versions of utilities explicitly, saying that we won't just support whatever POSIX-ish utilities your distro ships with, and won't support you unless you try GNU Coreutils and Bash first. I don't imagine that would go over well. Though I will be honest, if we could do that, it would make using shell a lot less painful. It's just not "the Unix way," if you know what I mean.

As a case in point... there a was a time when a certain person who used to work on this project kept wanting to replace pieces of Python with "simple POSIX sh scripts," as he called them, and it seemed like every other time he would do it, it would cause weird problems on obscure Linux distros that don't default to Bash, one of the BSDs, or SunOS. I never dared say this when he was around, but I kind of hated those shell scripts for that reason, and honestly that's what soured me on the idea of doing everything as shell scripts instead of Python. Because as bad as Python is/was... it wasn't as annoying as dealing with a random mix of tools from random vendors that are all named the same thing and all vaguely POSIX compliant but different in just enough ways that you can't trust them 100%. It also doesn't help that most people who think they know something is POSIX compliant are totally wrong and only managed to test on maybe two or three popular shells that happened to support an extension they were using. If there is one good thing about Python, that I have grudgingly come to appreciate from spending months on that port... it's that at least version differences are well-documented, and people complain loudly en masse about new versions breaking things all at once. So any problem we have with it, someone else has had before and likely complained. And that kind of research isn't as easy to do with implementations of sh and coreutils, though it certainly can be done, it's just buried a little deeper.

The only reason I mention Perl at all, is simply because I've found it to be a tool that is very conservative in behavior compared to Python, and also fairly consistent in behavior across platforms. I would also say that about GNU m4, which autoconf is written in. So the way I'm seeing it is... autoconf, perl, and GNU make are known, fairly conservative tools that are not changing fast. Python is reasonably consistent cross-platform, but changes really fast across versions. Shell is reasonably consistent across time, but varies a lot between platforms.

So yeah, I'm definitely not advocating for getting rid of autoconf or GNU make. I'm just saying... maybe I'm not eager to see more things that invoke a #!/bin/sh shebang in the tree and use a ton of utilities like grep, awk, or sed. I suppose it won't hurt anything to keep the battle-tested ones we have (I've never been a believer in change for the sake of change anyway), but I definitely think it's safe to say I don't think it's the right direction for new code.

But really, what I'm finding is that all the tools we use to build this have their strengths and weaknesses, and we just have to learn them well enough to find what works well in what contexts through trial and error. It should probably be less of a "replace X with Y," and more of a "replace this instance of X with this instance of Y, because of issues A, B, and C that are addressed by doing so." Like, let's say we run into an issue where different platform's shells or coreutils do different things... we could debug POSIX compliance. Or, in some cases, it would be easy to replace that shell script with a perl one-liner because perl works a lot like a shell with most coreutils integrated even though it's technically a scripting language. Or maybe there's a Python script that keeps getting hit by version change after version change, or Python just feels really sluggish to invoke for a simple task because it's such a heavy program, and eventually we decide to do that one thing in a Makefile, or an m4 macro, or a perl script, using whatever we deem the best tool for the job.