Intended archive format change for Linux

[Moonchild]

Considering all supported distros to the best of my knowledge have xz support for tarballs (although maybe not always installed by default), I'd like to change the Linux tarball archive format from bz2 to xz for smaller downloads and (on our side as well) less needed bandwidth. But, I'm unsure as to how the Linux users would view such a change, so I'm asking y'all: xz or bz2, or you don't really care either way? Please use the poll to make your choice -- no need to announce in replies what you've chosen, it's redundant.

Note: If your distro doesn't have xz support at all, please let us know, and mention which distro you are using. Please check first if it might be available but simply not installed.

[daftman1]

Why not use zstd? Arch Linux recently switched their package manager to use it instead of xz, they claim the encryption/decryption speeds are faster while compression level is comparable to xz.

[New Tobin Paradigm]

Because these are generic binaries intended to run on many systems and seeing as I have never heard of that format before today.. Who knows what all supports it. xz by now should be well supported by all distros and is deff supported in the majority of them.

I also don't think using Facebook technology is a good thing.

[Moonchild]

encryption/decryption speeds

No encryption is used so this is not an argument anyway.
Like Tobin I never even heard of this format before your mention of it, meaning it's likely hardly adopted anywhere, not mature, and certainly unsuited for software distribution to a broad audience. In addition, after 1 minute research, it's clear that it's not meant for archive compression, it's meant for stream/transport/etc. compression ("real-time"). Compression ratios are also not the same as LZMA (considerably less[1]) which makes it a non-starter for this purpose. Not sure why Arch would decide to switch to it in favor of xz for packages, unless they are suffering from Magpie Syndrome.

[1]I did some basic comparison with a typical data set (/dist/bin of a built Pale Moon). Zip = 41.4 MB; zstd (default setting at fast) = 36.6 MB; zstd (ultra) = 30.4 MB; 7z (ultra) = 27.6 MB -- of note is that compression speed of zstd at ultra was very slow (~1 MB/s) compared to 7z (~3.6 MB/s), on my system.

[daftman1]

Sorry I meant compression/decompression speeds (instead of encryption) but of course you are correct.
I took the statements from the Arch announcement as given while reality looks much less favourable for zstd - even speed doesn't seem as good as claimed (https://community.centminmod.com/thread ... etc.17259/).

[New Tobin Paradigm]

Considering that wasn't one of the options, I dunno why you suggested it.

[vannilla]

If I'm not mistaken, the Linux kernel itself started using XZ with version 5, or maybe even earlier.

[bgstack15]

No preference. Can use tar -J everywhere I've needed to. Thanks for checking with the users!

[Lunokhod]

Code: Select all

Original archive:
$ time tar -xjf 2019-11-11-palemoon-unstable-latest.linux-x86_64.tar.bz2 
real	0m4.879s
user	0m4.782s
sys	0m0.503s
Retar here for comparison in case of OS oddity
$ tar -cJf  xzpalemoon.tar.xz palemoon/
$ tar -cjf bpalemoon.tar.bz2 palemoon/
$ rm -r palemoon/
$ time tar -xJf xzpalemoon.tar.xz 
real	0m2.912s
user	0m2.790s
sys	0m0.579s
$ rm -r palemoon/
$ time tar -xjf bpalemoon.tar.bz2 
real	0m4.871s
user	0m4.801s
sys	0m0.476s

The bz2 is 47M while the xz is 40M, so that suggests both faster extraction and download times for xz. The concept works well here.

[New Tobin Paradigm]

When you consider using 7z on windows at ultra and xz on linux at max a single Pale Moon release can save upwards of 40 megs of server and archive storage.. 50 once GTK3 build is gotten around to. Of course smaller downloads mean less transfer both server and user side and it extracts faster than bz2.. Well that is a plus..

Good.

[loxodont]

I voted "no preference" as Pale Moon internal updates worked very well (since updater was dropped) and I guess if I need unpacking xz files my linuxes will do.

[Axatax]

Who cares?

In the age of symmetical gigabit fiber you can UUENCODE for all I care.

[Axatax]

Who cares?

In the age of symmetical gigabit fiber you can UUENCODE for all I care. It takes a whole 3 seconds to download.

[Moonchild]

In the age of symmetical gigabit fiber you can UUENCODE for all I care.

The ignorance displayed here is quite astonishing.

[stevenpusser]

Most distros should include parallel compressors for the various formats to take advantage of modern CPUs, but the only two I've used for xz are pxz and pixz.

[Moonchild]

Most distros should include parallel compressors for the various formats to take advantage of modern CPUs, but the only two I've used for xz are pxz and pixz.

I'm not entirely sure how this relates to the question at hand...

[bgstack15]

The user commented about various archiving tools used for the proposed archive format. That sounds relevant to me. Also, it might provide information to people who are not yet aware of such tools to open xz files. If you want a survey without general discussion, go use surveymonkey.

[Lucio Chiappetti]

While in general I'm not so keen in the "yet another compression tool" which each distro and version seem to use (compare e.g. the archiving of syslog and alike), it is easy to use a command like apropos xz to get all the commands needed. Or man xzcat if you dare to guess.

[karlchen]

Finding xz (XZ Utils) 5.1.0alpha on my older LTS systems (Ubuntu 16.04.6 / Mint 18.1), I decided to be a daredevil and opt for the xz format.

[Moonchild]

For the record, you don't need to "learn new tools" or read manuals for this change. you'd still be using good ol' tar to extract the tarball, just with a different flag (capital J) to indicate it's xz compressed. So you can get off of your proverbial soapboxes.

We won't be switching to anything else. xz/lzma has been tried and tested and is mature enough to consider at this point, that's why I posed the question. It's not supposed to be about any alternatives or how distros handle stuff internally when encountering the compressor or archiving tools or what not.

Off-topic:
And no, I won't be using any user-tracking big name survey engine that will Spam You To Death Forever After You've Used Them Once™ because they want to sell you something-as-a-service. All I ask is that everyone please keep discussion here ON TOPIC, i.e. directly related to the posed question in the original post. If y'all can't do that, then you risk more unilateral decisions in the future about the browser and its release engineering.