Consider pausing use of XZ, LZMA and 7z until the fallout settles

[jb_wisemo]

As almost everyone here is probably aware, a trusted upstream developer added a trojan horse to the upstream tarballs of LZMA sources in February but was found out during the Easter week, causing distros to rollback to an older version.

However liblzma is historically deeply intertwined with the 7z and xz compression formats, and there is no information yet about how deep in the upstream teams this goes.

Therefore, it would be prudent if upcoming Pale Moon and UXP releases reverted to older file formats ZIP 2.x and bzip2 for release files and any included compression/decompression code until the dust settles. Beware that some ZIP tools have included LZMA compression as an option, avoid those too.

[vannilla]

The exploit targets sshd, the payload explicitly tests for GCC, glibc and a Debian-based system (and I think Fedora too? I forgot) and more specifically those where sshd is patched to use hook into systemd so using xz or 7z to create a zip archive is hardly a issue. Additionally, only two versions (5.6.0 and 5.6.1) are affected, so using any other version is still feasible.

[Moonchild]

No, I'm not going to revert to zip/bz2 and upend my release engineering for this and cause upgrade compat issues.
If people feel it's "too much of a risk" someone in the community can always repackage the xz distributions for those people.

[Night Wing]

I do not know if this is related, but I run three linux distros and two of them are Debian based. Those two are, at this time, Debian 12.5 (Bookworm) Xfce and MX Linux 23.2 (Libretto) Xfce.

There was a discussion on the MX Forum site about this "breach of trust" concerning one of the developers who went rogue. About two days later, I received 14 security updates each for both MX Linux and Debian. As far as I know, these updates fixed the problem.

[Pentium4User]

The backdoor only affect special situations like when being linked by systemd/sshd.

Normal archive operations should be fine.
Please also check if the affected version was ever installed on your OS. If not, no risk.

[Night Wing]

MX Linux 23.2 and Debian 12.5 have been patched.

But this problem has created consequences because it is going to delay the release of Debian 12.6 (Bookworm) which was going to be released this upcoming Saturday, April 6th. There is no new date for this release as of right now.

https://linuxiac.com/debian-decided-to- ... 6-release/

[andyprough]

If some foreign (or domestic) government wants the data on my laptop, they won't have to spend years creating a multi-million dollar exploit of a compression algorithm. They can just buy a $5 wrench and hit me on the head with it until I give up my password.

[moonbat]

They can just buy a $5 wrench and hit me on the head with it until I give up my password.

I had this very xkcd printed up and pasted in my cubicle back when working on Java security.

[athenian200]

Honestly, I always kinda thought 7z was shady and seems a little too closely associated with malware and piracy, and wasn't a fan of switching to that on Windows for that reason... I don't imagine that association is going to improve in light of this news.

It's funny, I remember SmartScreen was always flagging stuff that was compressed with xz and 7z as potentially something that could be run by a remote attacker to gain control of the system. And mostly it was deemed a false positive upon human analysis due to that being a less common compression format that the heuristics didn't know how to deal with because, after all, the stuff inside the archive was found to be safe. What if... the heuristics were actually identifying the compression format itself as the threat and no one picked up on it? Probably a bit far-fetched, but you have to wonder in light of all this...

Anyway, MC is a security expert, so if he says it's fine, I'll believe him... I just still kinda wish we hadn't adopted 7z and xz now because of the associations more than because I think it's actually dangerous. It would be too much trouble to go back now, though, and also take up more space on the server. I'm half-tempted to cite it as another thing we can blame on you-know-who, but I don't think it would be fair to lay this one on his doorstep, even though he was the one that changed the build system to produce such archives...

For what it's worth, I've always used .zip for Epyrus on Windows, but this is actually not because of a fear of 7z, and is rather a convenience feature because Windows doesn't ship with a tool that can extract 7z by default. If distros start dropping xz as a package over this, I might compress the tarballs with bz2 again, but I would rather not have to do that because they will be larger in size. Still, if distros don't ship xz anymore, then it is what it is...

[Massacre]

Honestly, I always kinda thought 7z was shady and seems a little too closely associated with malware and piracy, and wasn't a fan of switching to that on Windows for that reason... I don't imagine that association is going to improve in light of this news.

Well, 7z is opensource, so there's much less problem with backdoors or exploits as in closed-source software, because of their better visibility. Exploits and backdoors could happen everytime and everywhere.

[moonbat]

Well, 7z is opensource

And xz isn't?
The whole reason why this kind of supply chain attack is disturbing is that despite it being open source and allegedly under the purview of thousands of eyeballs, someone still almost succeeded in adding malicious behavior to a widely used program.

[suzyne]

Honestly, I always kinda thought 7z was shady and seems a little too closely associated with malware and piracy

I don't know much about the history of the format, but isn't 7z only a compression algorithm that results in smaller files than zip? Specific executables (and from this story, I suppose sources) that compress or decompress a 7z file might be suspect, but not the idea itself.

[vannilla]

Snip

I think you should reconsider the way you view software.
Malware is distributed through ZIP files too; are you going to provide tar.gz archives for Windows, now? Except that malware has been distributed through tar files too, so I suppose that's out of the question too.
You should ignore these so-called allegations until someone actually complains about "using the same format as malware". Protection software does not count: how many times has Pale Moon been flagged as malware despite being a legitimate software? False positives happen regularly.

[Moonchild]

Anyway, MC is a security expert, so if he says it's fine, I'll believe him... I just still kinda wish we hadn't adopted 7z and xz now because of the associations more than because I think it's actually dangerous.

.7z and .xz are not dangerous. They are archive formats. The reality is that LZMA is simply a much better compression algorithm than LZ77 (used in ZIP) which is as you might guess quite dated.
I actually viewed an analysis by an expert when it came to light and the way the exploit works is both clever and extremely specific, and does not attack the format itself, but rather makes some very convoluted smart hooks that are very specific to ELF, systemd, and Linux binary handling on specific distros. As stated by others, it's a software supply chain issue of the compression library, not an issue with 7z/xz or the core of the lib itself. Ultimately, Open Source being maintained by many as a hobby is part of the issue when smart bad actors try to get away with something. Open Source being "auditable" isn't a guarantee it's actually also being audited or that things will get found...

If you want to get a fairly accessible explanation how it worked, check https://www.youtube.com/watch?v=jqjtNDtbDNI

[athenian200]

I think you should reconsider the way you view software.
Malware is distributed through ZIP files too; are you going to provide tar.gz archives for Windows, now? Except that malware has been distributed through tar files too, so I suppose that's out of the question too.
You should ignore these so-called allegations until someone actually complains about "using the same format as malware". Protection software does not count: how many times has Pale Moon been flagged as malware despite being a legitimate software? False positives happen regularly.

Well, to be clear, I myself don't think the software is dangerous except for the affected versions. Everything I'm seeing suggests it was tarballs packed by a specific maintainer (not the git repos), and even then only later versions, that have the malicious code. I'm more anxious about what the average person might think, than I am about it actually being an attack vector. Like, I would assume that if Linux distros start dropping xz-utils, it will be due to public perception rather than an actual security risk.

I'm mostly just listing off things that might cause the average person to be wary of the format, whether justified or not. You're absolutely right that it's not a valid association at all from a technical/programming perspective and that just not dealing with the affected versions of the library should be sufficient.

.7z and .xz are not dangerous. They are archive formats. The reality is that LZMA is simply a much better compression algorithm than LZ77 (used in ZIP) which is as you might guess quite dated.
I actually viewed an analysis by an expert when it came to light and the way the exploit works is both clever and extremely specific, and does not attack the format itself, but rather makes some very convoluted smart hooks that are very specific to ELF, systemd, and Linux binary handling on specific distros. As stated by others, it's a software supply chain issue of the compression library, not an issue with 7z/xz or the core of the lib itself. Ultimately, Open Source being maintained by many as a hobby is part of the issue when smart bad actors try to get away with something. Open Source being "auditable" isn't a guarantee it's actually also being audited or that things will get found...

If you want to get a fairly accessible explanation how it worked, check https://www.youtube.com/watch?v=jqjtNDtbDNI

That is a really good analysis, it goes straight to the source and everything. Summarizes stuff I had to watch 5 or 6 random videos to glean in a much nicer package.

Actually, the stuff cited about ELF, systemd, and Linux binary handling in general actually relates back to why I don't run Linux on bare metal in the first place, I was aware of some of this stuff and was vaguely uncomfortable because of it, but wasn't aware of its role in this specific exploit. It reminds me of the kind of ELF-related details I had to dig into to get libffi to compile on SunOS and get libxul to link properly, only used by someone a lot less ethical.

If anything, it's more an inditement of mainstream Linux distros and the way their package repos are maintained... though all the news headlines are more likely to make it about xz-utils and LZMA itself rather than about the underlying weaknesses in Linux as an OS that made it so easy to turn a random compression library into a backdoor. What's even funnier to me, is that it was only discovered because of a random Microsoft employee...

[ChrisCat]

If anything, it's more an inditement of mainstream Linux distros and the way their package repos are maintained... though all the news headlines are more likely to make it about xz-utils and LZMA itself rather than about the underlying weaknesses in Linux as an OS that made it so easy to turn a random compression library into a backdoor.

liblzma is hardly a "random compression library". LZMA is a highly used compression algorithm in computers, with liblzma being the primary library to handle it, so it's use will be pervasive on a given system, which in turn makes it a nice looking target for bad actors. It's like calling libX11 a "random gui library" for a bad actor to backdoor.

All software has their own attack vectors, so an attack on one will of course be tailored to what it's susceptible to and avoid what it's strong against. But even there, it doesn't work on all Linux systems because of the way packages are handled by various repos, and the way Linux systems can be configured; monocultures be damned. If you didn't have a patched sshd that caused liblzma to be linked into the process, you were fine, if you didn't use glibc, you were fine, if you didn't use a Debian-based build, you were fine. It's not like random DLLs on Windows haven't contained backdoors either, where the OS is more fixed to a particular configuration an attacker could rely on.

What's even funnier to me, is that it was only discovered because of a random Microsoft employee...

I find it more amusing it was discovered because someone noticed ssh login would take 0.8 seconds as opposed to 0.3, and they were able to dive into the source to figure out why along with the responsible party via the public commit logs. You wouldn't get that kind of response on Windows or Mac.

[athenian200]

liblzma is hardly a "random compression library". LZMA is a highly used compression algorithm in computers, with liblzma being the primary library to handle it, so it's use will be pervasive on a given system, which in turn makes it a nice looking target for bad actors. It's like calling libX11 a "random gui library" for a bad actor to backdoor.

Certainly didn't mean to suggest LZMA isn't an important algorithm, apologies if that's how it sounded. I was speaking specifically about liblzma as a library. What I mean is that liblzma specifically seems to be a poorly-maintained hobby project with only a handful of maintainers, but Linux distros employ it casually as if it were something as reliable as more established libraries like bz2 or gz. LZMA compression is indeed increasingly common, but if something as tiny and lacking oversight as liblzma is the primary tool people use for dealing with it, then that's worrying because they are using a compression library that probably shouldn't be taken very seriously to do real work, even linking it into an ssh daemon of all things. Not sure if "random" was the right word. Obscure? Tiny? Untested? Not sure what the right language to use here is.

I guess ultimately that's what all the "software supply chain" talk is about. The fact that small hobby projects are deployed as part of major pieces of infrastructure that people actually rely on. Maybe this incident will result in some major entity like Red Hat or Canonical forking the project, auditing all the code, and continuing development with a much larger team, who knows?

[ChrisCat]

Certainly didn't mean to suggest LZMA isn't an important algorithm, apologies if that's how it sounded. I was speaking specifically about liblzma as a library. What I mean is that liblzma specifically seems to be a poorly-maintained hobby project with only a handful of maintainers, but Linux distros employ it casually as if it were something as reliable as more established libraries like bz2 or gz.

Ah. Yeah, though that's not exactly a Linux distro specific thing:

Open source software tends to be a thankless job, showing itself to be incredibly useful with few people realizing just how few people are actually maintaining it. It's easy to think "important project = lots of help and donations to keep it working", but that's often not the case. Ideally, companies and people with resources would offer help to the projects they rely on, but most companies aren't in the business of charity, and people with resources either don't have enough to go around or don't know who's the most in need.

Not sure if "random" was the right word. Obscure? Tiny? Untested? Not sure what the right language to use here is.

I suppose "not well maintained" is the most accurate description. But as pointed out above, it's easy to overlook that something isn't well maintained.

Though it's also the case that not being well maintained itself wasn't the problem; small utility libraries like that don't really need all that much upkeep as they aren't getting new features all the time and there's no reason to change what works. So much of the work tends to be on bugfixes and subtle performance improvements, which are largely invisible to most users. Work on the project could've stopped wholesale and not many people would be bothered by it, existing releases won't suddenly break when development on new versions stop. But that lack of maintenance created an opening for an attacker to get into the project's good graces through social engineering (over multiple years), giving the attacker permission with the project they wouldn't have otherwise had.

[Echedelle]

As almost everyone here is probably aware, a trusted upstream developer added a trojan horse to the upstream tarballs of LZMA sources in February but was found out during the Easter week, causing distros to rollback to an older version.

However liblzma is historically deeply intertwined with the 7z and xz compression formats, and there is no information yet about how deep in the upstream teams this goes.

Therefore, it would be prudent if upcoming Pale Moon and UXP releases reverted to older file formats ZIP 2.x and bzip2 for release files and any included compression/decompression code until the dust settles. Beware that some ZIP tools have included LZMA compression as an option, avoid those too.

Even in the event of such issues, ignoring that you are mixing 7z with xz and the LZMA implementation of xz and that this only affects specific package versions... there is always the lzip implementation of LZMA x3

[athenian200]

Even in the event of such issues, ignoring that you are mixing 7z with xz and the LZMA implementation of xz and that this only affects specific package versions... there is always the lzip implementation of LZMA x3

Just looked into lzip, and it seems like it is indeed a separate implementation of LZMA that compresses about as well as xz does... so LZMA itself isn't the problem, it's the liblzma implementation itself.

Looking into this more, I just looked at the 7zip source code and I can't find any actual evidence of liblzma as a dependency.

It appears that the developer of 7zip (Igor Pavlov) did contribute to the development of xz-utils at some point, but looking at the 7zip source code, it doesn't appear that liblzma is actually a build dependency for 7-zip and I can't find the name of Lasse Collin anywhere in the source code that 7zip uses. Also, he distributes his own lzma-sdk as public domain software, and I'm starting to wonder if Lasse Collin used his code as a base for xz-utils rather than the other way around, if any code was shared.

So are we actually sure that 7z even uses the same liblzma library as xz? I think people may have assumed this based on circumstantial evidence, seeing that Igor Pavlov's name is associated with liblzma, and there is a link back to XZ-Utils on the LZMA-SDK website, but it appears that 7zip may actually have its own independent LZMA stuff going on. Here's the line that makes me think that...

ANSI-C and C++ source code in LZMA SDK is subset of source code of 7-Zip.

So it seems that 7zip has existed longer than xz-utils, and may not depend on the Lasse Collin/Jia Tan implementation of liblzma at all? If that's the case, then it looks like everyone jumped to conclusions... while there was a arbitrary code execution issue in some older versions of 7zip self-extractor, that was over 5 years ago and mostly involved exploiting how Windows DLLs work rather than anything inherent in 7zip itself.

I'm not actually sure that there is that much cause for concern, even with liblzma now that the person responsible is off the project, but even if there was, it appears the use of it may be less widespread outside of Linux than previously thought. In other words, liblzma isn't the only library you can use to compress/decompress with LZMA, but Linux distros for whatever reason don't use the alternatives much on average.