Intended archive format change for Linux
Moderator: trava90
Forum rules
This board is for technical/general usage questions and troubleshooting for the Pale Moon browser only.
Technical issues and questions not related to the Pale Moon browser should be posted in other boards!
Please keep off-topic and general discussion out of this board, thank you!
This board is for technical/general usage questions and troubleshooting for the Pale Moon browser only.
Technical issues and questions not related to the Pale Moon browser should be posted in other boards!
Please keep off-topic and general discussion out of this board, thank you!
-
- Astronaut
- Posts: 652
- Joined: 2015-07-30, 20:29
- Location: Vaughan, ON, Canada
Re: Intended archive format change for Linux
I checked on the Puppy linux Pale Moon thread. It looks like all Puppy versions that "officially" support Pale Moon 28.7.x have xz support. There seems to be one extreme edge case of a really old version without xz support, that can be coaxed to run Pale Moon 28.7.x with some tweaking. But if they're going to backport glibc to it, they can probably backport xz. xz support looks OK on Puppy linux, so no problem there.
There's a right way
There's a wrong way
And then there's my way
There's a wrong way
And then there's my way
Re: Intended archive format change for Linux
Thank you for your entirely pointless report Dnes. Now crawl back into your bottle and go home please.
-
- Pale Moon guru
- Posts: 35649
- Joined: 2011-08-28, 17:27
- Location: Motala, SE
Re: Intended archive format change for Linux
Can we please keep unnecessary personal jabs out of this thread, no matter how irrelevant the offered information is?
Thank you.
Thank you.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
-
- Project Contributor
- Posts: 903
- Joined: 2015-08-01, 18:33
Re: Intended archive format change for Linux
It was more a suggestion for those creating the xz files, not for the end users. I also create xz files for general distribution, and wanted to pass on a tip.Moonchild wrote: ↑2019-11-17, 23:27I'm not entirely sure how this relates to the question at hand...stevepusser wrote: ↑2019-11-17, 22:22Most distros should include parallel compressors for the various formats to take advantage of modern CPUs, but the only two I've used for xz are pxz and pixz.
I remember very conservative Debian switched to xz compression as default for their deb packages way back in 2011, so they considered it stable enough way back then. Pale Moon won't even build on the two Debian releases that came after that change, Lenny and Squeeze, since their gcc versions are too old.
-
- Lunatic
- Posts: 469
- Joined: 2017-04-20, 21:25
Re: Intended archive format change for Linux
pixz doesn't seem to help unless multithreaded compression is used. Also using higher compression levels does not seem to help extraction speeds. So there may be an optimum way to compress with xz, (like using say -T 32) which currently can do multi threaded compression but not decompression. Weirdly both xz and pixz only ever use one of my 2 CPU's according to top. Perhaps that's because they are being run via time & tar? There wasn't a big difference in file size between 6 (default) or 9e (max) but 9 meant pixz extracted at speeds similar to a single thread. I wonder if these multi thread archives would be compatible with old versions of xz though?
Code: Select all
$ time tar -I "xz -6" -cf 1-6xzpalemoon.tar.xz palemoon/
real 1m1.560s
user 1m1.332s
sys 0m0.793s
$ time tar -I "xz -6 -T 2" -cf 2-6xzpalemoon.tar.xz palemoon/
real 0m36.787s
user 1m6.327s
sys 0m0.592s
$ time tar -I "xz -6 -T 32" -cf 32-6xzpalemoon.tar.xz palemoon/
real 0m35.816s
user 1m10.004s
sys 0m0.741s
$ time tar -I "pixz -9e -p 2" -cf pixz-p2-9e-xzpalemoon.tar.xz palemoon/
real 1m18.553s
user 1m17.928s
sys 0m0.760s
$ time tar -I "pixz -p 2" -xf 1-6xzpalemoon.tar.xz
real 0m3.023s
user 0m2.744s
sys 0m0.428s
$ time tar -I "pixz -p 2" -xf 2-6xzpalemoon.tar.xz
real 0m1.813s
user 0m2.844s
sys 0m0.431s
$ time tar -I "pixz -p 2" -xf 32-6xzpalemoon.tar.xz
real 0m1.797s
user 0m2.847s
sys 0m0.429s
$ time tar -I "pixz -p 2" -xf pixz-p2-9e-xzpalemoon.tar.xz
real 0m3.038s
user 0m2.713s
sys 0m0.465s
41065336 1-6xzpalemoon.tar.xz
40673208 pixz-p2-9e-xzpalemoon.tar.xz
Wait, it's all Ohio? Always has been...
-
- Moon Magic practitioner
- Posts: 2194
- Joined: 2018-05-05, 13:29
Re: Intended archive format change for Linux
De/compression speed, from what I understood, isn't really the problem here, but rather how xz creates a smaller archive compared to bzip2, so there's really no point in benchmarking different programs doing the same thing.
-
- Pale Moon guru
- Posts: 35649
- Joined: 2011-08-28, 17:27
- Location: Motala, SE
Re: Intended archive format change for Linux
Thank you, vanilla.
This is exactly it. Compression speed when building the archives is irrelevant. Decompression speeds on the user's system when installing is also irrelevant (unless of course it would be extremely slow -- but that's hardly ever the case, even with corner-case super-slow compressors). What -is- relevant here is download size. Shaving off a good 20-30% on each download adds up real quick when you are talking about TBs of bandwidth with a release.
This is exactly it. Compression speed when building the archives is irrelevant. Decompression speeds on the user's system when installing is also irrelevant (unless of course it would be extremely slow -- but that's hardly ever the case, even with corner-case super-slow compressors). What -is- relevant here is download size. Shaving off a good 20-30% on each download adds up real quick when you are talking about TBs of bandwidth with a release.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Intended archive format change for Linux
For my needs when I perposed this was more about storage than transfer but of course transfer does indeed help. If you consider there are 4 archives per application and each release at the maximum compression can save upwards of 40 megs and I intend to have two applications that is nearly 80 megs of server space I can save just by using 7z and xz it become significant over time for me.
-
- Lunatic
- Posts: 469
- Joined: 2017-04-20, 21:25
Re: Intended archive format change for Linux
Using standard xz, if you specify -T and use multithreaded compression - included in xz since 2014 - then you build archives the same size, but they build faster, and are compatible with multithreaded extraction if anyone was using that, or it was added to xz in future. Except in practice it didn't seem to work for max 9e compression, only with the default settings. Using tar -I "xz -6 -T 1" cf is exactly the same as tar -cJf which gives the default, it can be tuned in various ways but still uses xz and makes an xz archive.
Wait, it's all Ohio? Always has been...
-
- Pale Moon guru
- Posts: 35649
- Joined: 2011-08-28, 17:27
- Location: Motala, SE
Re: Intended archive format change for Linux
It's not relevant to this thread. Users won't be creating archives, they would be consuming them.vingtzwanzig wrote: ↑2019-11-20, 19:44Using standard xz, if you specify -T and use multithreaded compression {...}
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
-
- Lunatic
- Posts: 469
- Joined: 2017-04-20, 21:25
Re: Intended archive format change for Linux
"users consuming" agree 100%! Smaller = faster downloads, multithreaded creation = allows faster extraction (in some circumstances, where the user can utilize this). Probably makes little difference running manually, but if part of some automated package manager update it all helps. Still using straight "xz" and "on topic" But no matter, it's a minor detail. Whatever you decide upon will surely be marvellous. (Got an email notification hence the absurdly fast reply, just going to turn those off.)
Wait, it's all Ohio? Always has been...
-
- Board Warrior
- Posts: 1651
- Joined: 2018-06-08, 17:02
Re: Intended archive format change for Linux
Off-topic:
I had this in my browser history, & the article is dated, so, fwiw.
Xz format inadequate for long-term archiving
I had this in my browser history, & the article is dated, so, fwiw.
Xz format inadequate for long-term archiving
Re: Intended archive format change for Linux
Yeah competing format creator cries dot html. k
-
- Knows the dark side
- Posts: 4983
- Joined: 2015-12-09, 15:45
Re: Intended archive format change for Linux
Multiply that by the number of times the archive is downloaded, genius.
"One hosts to look them up, one DNS to find them and in the darkness BIND them."
Linux Mint 21 Xfce x64 on HP i5-5200 laptop, 12 GB RAM.
AutoPageColor|PermissionsPlus|PMPlayer|Pure URL|RecordRewind|TextFX
Linux Mint 21 Xfce x64 on HP i5-5200 laptop, 12 GB RAM.
AutoPageColor|PermissionsPlus|PMPlayer|Pure URL|RecordRewind|TextFX
Re: Intended archive format change for Linux
Why don't you both read the thread properly, dumbasses.
Compared to standard packaging in zip and bzip2 respectively 7z and xz both save 8-10 megabytes PER archive.. So Pale Moon for instance can save at most 40 megs for win32, win64, linux32, and linux64 and add ~10 megs for linux64-gtk3 in the future. That is 40 megabytes per complete release saved. It is significant enough to be worthwhile to code an alternate packaging scheme for our applications. It does add up. I have only said it repeatedly in this very thread.
SINCE the poll has a CLEAR majority of those who want this or don't care either way.. This is almost certainly going to happen. But there is one week left I suppose so anything could theoretically happen. Still, this poll of course only counts for Pale Moon and Basilisk. I am gonna do this in any event for the BinOC applications because I don't allow you people the chance to express your complete lack of understanding without consequence. My stuff follows my vision and my vision only. If you don't like then don't use it. So there.
Compared to standard packaging in zip and bzip2 respectively 7z and xz both save 8-10 megabytes PER archive.. So Pale Moon for instance can save at most 40 megs for win32, win64, linux32, and linux64 and add ~10 megs for linux64-gtk3 in the future. That is 40 megabytes per complete release saved. It is significant enough to be worthwhile to code an alternate packaging scheme for our applications. It does add up. I have only said it repeatedly in this very thread.
SINCE the poll has a CLEAR majority of those who want this or don't care either way.. This is almost certainly going to happen. But there is one week left I suppose so anything could theoretically happen. Still, this poll of course only counts for Pale Moon and Basilisk. I am gonna do this in any event for the BinOC applications because I don't allow you people the chance to express your complete lack of understanding without consequence. My stuff follows my vision and my vision only. If you don't like then don't use it. So there.
-
- Lunatic
- Posts: 469
- Joined: 2017-04-20, 21:25
Re: Intended archive format change for Linux
Off-topic:
lrzip is written by Con Kolivas who does the CK kernel, and lrztar makes it easy to use / type in the terminal. It used both my CPU's fully. But decompression of -z takes nearly as long as compression. ZPAQ max saves almost 12MB from the current bz2! (It also has other compression methods.)
zstd is fast and uses both CPU's but isn't that great on size. (Fedora are adopting this for packages too)
xz is good overall on size and speed. Parallel processing seems a WIP for now.
Code: Select all
51604716 tar --zstd -cf (zstd default)
48945972 palemoon-unstable-latest.linux-x86_64.tar.bz2 (original download)
48111866 tar -I "lrzip -b -L9" (lrzip using bz2 at max compression)
43479395 tar -I "zstd --ultra -T2 -22" -cf (zstd at max compression)
41065336 tar -I "xz" -cf (xz default)
40758594 tar -I "lzip -9" -cf (lzip max compression)
40672388 tar -I "xz -9e" -cf (xz max compression)
40064748 tar -I "lrzip -z" (equivalent to lrztar -z) (lrzip ZPAQ default)
37080633 lrztar -z -L9 (lrzip ZPAQ max)
zstd is fast and uses both CPU's but isn't that great on size. (Fedora are adopting this for packages too)
xz is good overall on size and speed. Parallel processing seems a WIP for now.
Wait, it's all Ohio? Always has been...
Re: Intended archive format change for Linux
xz is also mature and seems supported everywhere and is installed by default or easy to get from package managers in addition to being the best over all for size and speed.
And it isn't developed by a crybaby or evil Facebook.. So that is an additional bonus.
May I remind everyone that the poll was created to present either current bz2 OR xz.
And it isn't developed by a crybaby or evil Facebook.. So that is an additional bonus.
May I remind everyone that the poll was created to present either current bz2 OR xz.
-
- Pale Moon guru
- Posts: 35649
- Joined: 2011-08-28, 17:27
- Location: Motala, SE
Re: Intended archive format change for Linux
Please stop tossing in alternative archive formats. read the original post.
If you still insist on generating noise with more unrelated archive formats, compression methods, tools, and/or other content to this thread, you'll have to deal with the consequences.
Focus, people. Focus.
If you still insist on generating noise with more unrelated archive formats, compression methods, tools, and/or other content to this thread, you'll have to deal with the consequences.
Focus, people. Focus.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite