Salvaging add-ons from AMO

Anything to do with the Pale Moon add-ons website. (addons.palemoon.org)
Not for questions about add-ons themselves!
Forum rules
Important: This board is for specifics regarding the add-ons website (addons.palemoon.org) and not to report extension compatibility issues or discuss different extensions.
Please only post here when your topic is directly related to the add-ons website service so our moderators don't have to move your posts all the time...
__Vano

Salvaging add-ons from AMO

Unread post by __Vano » 2016-06-06, 15:37

Due to Mozilla deprecating XUL add-ons, they might decide to remove XUL versions from the site altogether after they do that as "no longer relevant".

In that case, we need to salvage PaleMoon-usable versions for the ones being used by the community to APO before that happens. To decide which ones to take, PaleMoon should probably come with a preinstalled extension to report extensions from AMO in use (anonymously, of course, and the user should be asked before sending, clearly explaining the reason for collection).

Is this worth considering or am I just being paranoid? :)

New Tobin Paradigm

Re: Salvaging add-ons from AMO

Unread post by New Tobin Paradigm » 2016-06-06, 15:59

Not an unfounded potentiality .. I have already done this once and have a full copy of AMO as of late 2014 and have done much magic to pull out and update with last known compatible version for Pale Moon as of sometime late 2015 based on what extensions users actually use.

This will need to be ran again in the near future and also for the targets of 28 and 38 and is on my agenda. But no need for some reporting extension I can do this at the server level with the AUS script.

Process involves putting a tap on AUS and saving the GUID/ID of extensions when the request comes in.. But ONLY the guid because all other info is unimportant. Then we simply query AMO's AUS and pull xpi files down no need for the user to do anything or deal with it.

Basically it is a manual process of what normally happens automagically but on a large scale. I do hate doing this though because it has to be left to run for at least 48 hours and then have to moderate the requests to AMO so it doesn't create the appearance of some attack doing thousands of queries from a single location. As for the Pale Moon user.. Like I said.. Just need to collect GUIDs..

I don't believe in logging statistical information as a matter of course because it would not only be a lot of data to manage and parse but those files grow large very quickly and I can't be bothered to deal with it normal circumstances. Any sort of logging on our side (for the add-ons server) is only enabled and used when there might be a problem (trying to fix something broken) or something unusual like this exact scenario and once acted upon the data is destroyed and whatever mechanism I turned on is removed or shut off.

Klappis

Re: Salvaging add-ons from AMO

Unread post by Klappis » 2016-07-16, 18:15

Matt A Tobin wrote:Not an unfounded potentiality .. I have already done this once and have a full copy of AMO as of late 2014 and have done much magic to pull out and update with last known compatible version for Pale Moon as of sometime late 2015 based on what extensions users actually use.

This will need to be ran again in the near future and also for the targets of 28 and 38 and is on my agenda. But no need for some reporting extension I can do this at the server level with the AUS script.

Process involves putting a tap on AUS and saving the GUID/ID of extensions when the request comes in.. But ONLY the guid because all other info is unimportant. Then we simply query AMO's AUS and pull xpi files down no need for the user to do anything or deal with it.

Basically it is a manual process of what normally happens automagically but on a large scale. I do hate doing this though because it has to be left to run for at least 48 hours and then have to moderate the requests to AMO so it doesn't create the appearance of some attack doing thousands of queries from a single location. As for the Pale Moon user.. Like I said.. Just need to collect GUIDs..

I don't believe in logging statistical information as a matter of course because it would not only be a lot of data to manage and parse but those files grow large very quickly and I can't be bothered to deal with it normal circumstances. Any sort of logging on our side (for the add-ons server) is only enabled and used when there might be a problem (trying to fix something broken) or something unusual like this exact scenario and once acted upon the data is destroyed and whatever mechanism I turned on is removed or shut off.
How can i download all xpi files at once?

New Tobin Paradigm

Re: Salvaging add-ons from AMO

Unread post by New Tobin Paradigm » 2016-07-17, 00:56

You .. can't. They killed that when they killed their actual FTP server and the "cloud mirror" that sits on ftp.mozilla.org which is only html now does not have it. That is unless you want to query and parse metadata and attempt download of every file by brute force. But expect to get your IP blocked for network abuse.

dark_moon

Re: Salvaging add-ons from AMO

Unread post by dark_moon » 2016-07-17, 07:50

You can load the whole AMO website without getting IP ban with https://httrack.com
I do this months before too.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35481
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Salvaging add-ons from AMO

Unread post by Moonchild » 2016-07-20, 12:32

dark_moon wrote:You can load the whole AMO website without getting IP ban with https://httrack.com
I do this months before too.
httrack crawls and downloads in rapid succession -- if you didn't get blocked doing that then you were lucky.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Falna
Astronaut
Astronaut
Posts: 511
Joined: 2015-08-23, 17:56
Location: UK / France

Re: Salvaging add-ons from AMO

Unread post by Falna » 2016-07-20, 17:57

Moonchild wrote:httrack crawls and downloads in rapid succession
...unless you change the settings to do so slowly - which is a good idea.

Forked extensions :
● Add-ons Inspector ● Auto Text Link ● Copy As Plain Text ● Copy Hyperlink Text ● FireFTP button replacement ● gSearch Bar ● Navigation Bar Enhancer ● New Tab Links ● Number Tabs ● Print Preview Button and Keyboard Shortcut 2 ● Scrollbar Search Marker ● Simple Marker ● Tabs To Portfolio ● Update Alert ● Web Developer's Toolbox ● Zap Anything

Hint: If you expect a reply to your PM, allow replies...

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35481
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Salvaging add-ons from AMO

Unread post by Moonchild » 2016-07-21, 08:53

If you want to let it run for days on end, sure ;) and even then, those usage patterns may get you blocked.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

dark_moon

Re: Salvaging add-ons from AMO

Unread post by dark_moon » 2016-07-21, 10:01

Many months ago i load whole AMO with maximum speed settings and don't get any blocked stuff.
Maybe they change that now but why should they care?

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35481
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Salvaging add-ons from AMO

Unread post by Moonchild » 2016-07-21, 12:34

It's considered network abuse if you make an unfair amount of requests (not normal browsing behavior) and/or use up a disproportionate amount of bandwidth.
I'm surprised you don't realize this. It's common sense.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked