Intended archive format change for Linux

[Walter Dnes]

I checked on the Puppy linux Pale Moon thread. It looks like all Puppy versions that "officially" support Pale Moon 28.7.x have xz support. There seems to be one extreme edge case of a really old version without xz support, that can be coaxed to run Pale Moon 28.7.x with some tweaking. But if they're going to backport glibc to it, they can probably backport xz. xz support looks OK on Puppy linux, so no problem there.

[New Tobin Paradigm]

Thank you for your entirely pointless report Dnes. Now crawl back into your bottle and go home please.

[Moonchild]

Can we please keep unnecessary personal jabs out of this thread, no matter how irrelevant the offered information is?

Thank you.

[stevenpusser]

Most distros should include parallel compressors for the various formats to take advantage of modern CPUs, but the only two I've used for xz are pxz and pixz.

I'm not entirely sure how this relates to the question at hand...

It was more a suggestion for those creating the xz files, not for the end users. I also create xz files for general distribution, and wanted to pass on a tip.

I remember very conservative Debian switched to xz compression as default for their deb packages way back in 2011, so they considered it stable enough way back then. Pale Moon won't even build on the two Debian releases that came after that change, Lenny and Squeeze, since their gcc versions are too old.

[Lunokhod]

pixz doesn't seem to help unless multithreaded compression is used. Also using higher compression levels does not seem to help extraction speeds. So there may be an optimum way to compress with xz, (like using say -T 32) which currently can do multi threaded compression but not decompression. Weirdly both xz and pixz only ever use one of my 2 CPU's according to top. Perhaps that's because they are being run via time & tar? There wasn't a big difference in file size between 6 (default) or 9e (max) but 9 meant pixz extracted at speeds similar to a single thread. I wonder if these multi thread archives would be compatible with old versions of xz though?

Code: Select all

$  time tar -I "xz -6" -cf 1-6xzpalemoon.tar.xz palemoon/
real	1m1.560s
user	1m1.332s
sys	0m0.793s
$  time tar -I "xz -6 -T 2" -cf 2-6xzpalemoon.tar.xz palemoon/
real	0m36.787s
user	1m6.327s
sys	0m0.592s
$  time tar -I "xz -6 -T 32" -cf 32-6xzpalemoon.tar.xz palemoon/
real	0m35.816s
user	1m10.004s
sys	0m0.741s
$  time tar -I "pixz -9e -p 2" -cf pixz-p2-9e-xzpalemoon.tar.xz palemoon/
real	1m18.553s
user	1m17.928s
sys	0m0.760s


$  time tar -I "pixz -p 2" -xf 1-6xzpalemoon.tar.xz 
real	0m3.023s
user	0m2.744s
sys	0m0.428s
$  time tar -I "pixz -p 2" -xf 2-6xzpalemoon.tar.xz 
real	0m1.813s
user	0m2.844s
sys	0m0.431s
$  time tar -I "pixz -p 2" -xf 32-6xzpalemoon.tar.xz 
real	0m1.797s
user	0m2.847s
sys	0m0.429s
$  time tar -I "pixz -p 2" -xf pixz-p2-9e-xzpalemoon.tar.xz 
real	0m3.038s
user	0m2.713s
sys	0m0.465s

41065336 1-6xzpalemoon.tar.xz
40673208 pixz-p2-9e-xzpalemoon.tar.xz

[vannilla]

De/compression speed, from what I understood, isn't really the problem here, but rather how xz creates a smaller archive compared to bzip2, so there's really no point in benchmarking different programs doing the same thing.

[Moonchild]

Thank you, vanilla.

This is exactly it. Compression speed when building the archives is irrelevant. Decompression speeds on the user's system when installing is also irrelevant (unless of course it would be extremely slow -- but that's hardly ever the case, even with corner-case super-slow compressors). What -is- relevant here is download size. Shaving off a good 20-30% on each download adds up real quick when you are talking about TBs of bandwidth with a release.

[New Tobin Paradigm]

For my needs when I perposed this was more about storage than transfer but of course transfer does indeed help. If you consider there are 4 archives per application and each release at the maximum compression can save upwards of 40 megs and I intend to have two applications that is nearly 80 megs of server space I can save just by using 7z and xz it become significant over time for me.

[Lunokhod]

Using standard xz, if you specify -T and use multithreaded compression - included in xz since 2014 - then you build archives the same size, but they build faster, and are compatible with multithreaded extraction if anyone was using that, or it was added to xz in future. Except in practice it didn't seem to work for max 9e compression, only with the default settings. Using tar -I "xz -6 -T 1" cf is exactly the same as tar -cJf which gives the default, it can be tuned in various ways but still uses xz and makes an xz archive.

[Moonchild]

Using standard xz, if you specify -T and use multithreaded compression {...}

It's not relevant to this thread. Users won't be creating archives, they would be consuming them.

[Lunokhod]

"users consuming" agree 100%! Smaller = faster downloads, multithreaded creation = allows faster extraction (in some circumstances, where the user can utilize this). Probably makes little difference running manually, but if part of some automated package manager update it all helps. Still using straight "xz" and "on topic" But no matter, it's a minor detail. Whatever you decide upon will surely be marvellous. (Got an email notification hence the absurdly fast reply, just going to turn those off.)

[therube]

Off-topic:
I had this in my browser history, & the article is dated, so, fwiw.
Xz format inadequate for long-term archiving

[New Tobin Paradigm]

Yeah competing format creator cries dot html. k

[Axatax]

In the age of symmetical gigabit fiber you can UUENCODE for all I care.

The ignorance displayed here is quite astonishing.

Why? Is xz going to save you 300K over bzip2? It's splitting hairs at this point.

[moonbat]

Why? Is xz going to save you 300K over bzip2? It's splitting hairs at this point.

Multiply that by the number of times the archive is downloaded, genius.

[New Tobin Paradigm]

Why don't you both read the thread properly, dumbasses.

Compared to standard packaging in zip and bzip2 respectively 7z and xz both save 8-10 megabytes PER archive.. So Pale Moon for instance can save at most 40 megs for win32, win64, linux32, and linux64 and add ~10 megs for linux64-gtk3 in the future. That is 40 megabytes per complete release saved. It is significant enough to be worthwhile to code an alternate packaging scheme for our applications. It does add up. I have only said it repeatedly in this very thread.

SINCE the poll has a CLEAR majority of those who want this or don't care either way.. This is almost certainly going to happen. But there is one week left I suppose so anything could theoretically happen. Still, this poll of course only counts for Pale Moon and Basilisk. I am gonna do this in any event for the BinOC applications because I don't allow you people the chance to express your complete lack of understanding without consequence. My stuff follows my vision and my vision only. If you don't like then don't use it. So there.

[Lunokhod]

Off-topic:

Code: Select all

51604716 tar --zstd -cf                                   (zstd default)
48945972 palemoon-unstable-latest.linux-x86_64.tar.bz2    (original download)
48111866 tar -I "lrzip -b -L9"                            (lrzip using bz2 at max compression)
43479395 tar -I "zstd --ultra -T2 -22" -cf                (zstd at max compression)
41065336 tar -I "xz" -cf                                  (xz default)
40758594 tar -I "lzip -9" -cf                             (lzip max compression)
40672388 tar -I "xz -9e" -cf                              (xz max compression)
40064748 tar -I "lrzip -z" (equivalent to lrztar -z)      (lrzip ZPAQ default)
37080633 lrztar -z -L9                                    (lrzip ZPAQ max)

lrzip is written by Con Kolivas who does the CK kernel, and lrztar makes it easy to use / type in the terminal. It used both my CPU's fully. But decompression of -z takes nearly as long as compression. ZPAQ max saves almost 12MB from the current bz2! (It also has other compression methods.)
zstd is fast and uses both CPU's but isn't that great on size. (Fedora are adopting this for packages too)
xz is good overall on size and speed. Parallel processing seems a WIP for now.

[New Tobin Paradigm]

xz is also mature and seems supported everywhere and is installed by default or easy to get from package managers in addition to being the best over all for size and speed.

And it isn't developed by a crybaby or evil Facebook.. So that is an additional bonus.

May I remind everyone that the poll was created to present either current bz2 OR xz.

[Moonchild]

Please stop tossing in alternative archive formats. read the original post.
If you still insist on generating noise with more unrelated archive formats, compression methods, tools, and/or other content to this thread, you'll have to deal with the consequences.

Focus, people. Focus.