Salvaging add-ons from AMO
Forum rules
Important: This board is for specifics regarding the add-ons website (addons.palemoon.org) and not to report extension compatibility issues or discuss different extensions.
Please only post here when your topic is directly related to the add-ons website service so our moderators don't have to move your posts all the time...
Important: This board is for specifics regarding the add-ons website (addons.palemoon.org) and not to report extension compatibility issues or discuss different extensions.
Please only post here when your topic is directly related to the add-ons website service so our moderators don't have to move your posts all the time...
Salvaging add-ons from AMO
Due to Mozilla deprecating XUL add-ons, they might decide to remove XUL versions from the site altogether after they do that as "no longer relevant".
In that case, we need to salvage PaleMoon-usable versions for the ones being used by the community to APO before that happens. To decide which ones to take, PaleMoon should probably come with a preinstalled extension to report extensions from AMO in use (anonymously, of course, and the user should be asked before sending, clearly explaining the reason for collection).
Is this worth considering or am I just being paranoid?
In that case, we need to salvage PaleMoon-usable versions for the ones being used by the community to APO before that happens. To decide which ones to take, PaleMoon should probably come with a preinstalled extension to report extensions from AMO in use (anonymously, of course, and the user should be asked before sending, clearly explaining the reason for collection).
Is this worth considering or am I just being paranoid?
Re: Salvaging add-ons from AMO
Not an unfounded potentiality .. I have already done this once and have a full copy of AMO as of late 2014 and have done much magic to pull out and update with last known compatible version for Pale Moon as of sometime late 2015 based on what extensions users actually use.
This will need to be ran again in the near future and also for the targets of 28 and 38 and is on my agenda. But no need for some reporting extension I can do this at the server level with the AUS script.
Process involves putting a tap on AUS and saving the GUID/ID of extensions when the request comes in.. But ONLY the guid because all other info is unimportant. Then we simply query AMO's AUS and pull xpi files down no need for the user to do anything or deal with it.
Basically it is a manual process of what normally happens automagically but on a large scale. I do hate doing this though because it has to be left to run for at least 48 hours and then have to moderate the requests to AMO so it doesn't create the appearance of some attack doing thousands of queries from a single location. As for the Pale Moon user.. Like I said.. Just need to collect GUIDs..
I don't believe in logging statistical information as a matter of course because it would not only be a lot of data to manage and parse but those files grow large very quickly and I can't be bothered to deal with it normal circumstances. Any sort of logging on our side (for the add-ons server) is only enabled and used when there might be a problem (trying to fix something broken) or something unusual like this exact scenario and once acted upon the data is destroyed and whatever mechanism I turned on is removed or shut off.
This will need to be ran again in the near future and also for the targets of 28 and 38 and is on my agenda. But no need for some reporting extension I can do this at the server level with the AUS script.
Process involves putting a tap on AUS and saving the GUID/ID of extensions when the request comes in.. But ONLY the guid because all other info is unimportant. Then we simply query AMO's AUS and pull xpi files down no need for the user to do anything or deal with it.
Basically it is a manual process of what normally happens automagically but on a large scale. I do hate doing this though because it has to be left to run for at least 48 hours and then have to moderate the requests to AMO so it doesn't create the appearance of some attack doing thousands of queries from a single location. As for the Pale Moon user.. Like I said.. Just need to collect GUIDs..
I don't believe in logging statistical information as a matter of course because it would not only be a lot of data to manage and parse but those files grow large very quickly and I can't be bothered to deal with it normal circumstances. Any sort of logging on our side (for the add-ons server) is only enabled and used when there might be a problem (trying to fix something broken) or something unusual like this exact scenario and once acted upon the data is destroyed and whatever mechanism I turned on is removed or shut off.
Re: Salvaging add-ons from AMO
How can i download all xpi files at once?Matt A Tobin wrote:Not an unfounded potentiality .. I have already done this once and have a full copy of AMO as of late 2014 and have done much magic to pull out and update with last known compatible version for Pale Moon as of sometime late 2015 based on what extensions users actually use.
This will need to be ran again in the near future and also for the targets of 28 and 38 and is on my agenda. But no need for some reporting extension I can do this at the server level with the AUS script.
Process involves putting a tap on AUS and saving the GUID/ID of extensions when the request comes in.. But ONLY the guid because all other info is unimportant. Then we simply query AMO's AUS and pull xpi files down no need for the user to do anything or deal with it.
Basically it is a manual process of what normally happens automagically but on a large scale. I do hate doing this though because it has to be left to run for at least 48 hours and then have to moderate the requests to AMO so it doesn't create the appearance of some attack doing thousands of queries from a single location. As for the Pale Moon user.. Like I said.. Just need to collect GUIDs..
I don't believe in logging statistical information as a matter of course because it would not only be a lot of data to manage and parse but those files grow large very quickly and I can't be bothered to deal with it normal circumstances. Any sort of logging on our side (for the add-ons server) is only enabled and used when there might be a problem (trying to fix something broken) or something unusual like this exact scenario and once acted upon the data is destroyed and whatever mechanism I turned on is removed or shut off.
Re: Salvaging add-ons from AMO
You .. can't. They killed that when they killed their actual FTP server and the "cloud mirror" that sits on ftp.mozilla.org which is only html now does not have it. That is unless you want to query and parse metadata and attempt download of every file by brute force. But expect to get your IP blocked for network abuse.
Re: Salvaging add-ons from AMO
You can load the whole AMO website without getting IP ban with https://httrack.com
I do this months before too.
I do this months before too.
Re: Salvaging add-ons from AMO
httrack crawls and downloads in rapid succession -- if you didn't get blocked doing that then you were lucky.dark_moon wrote:You can load the whole AMO website without getting IP ban with https://httrack.com
I do this months before too.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Salvaging add-ons from AMO
...unless you change the settings to do so slowly - which is a good idea.Moonchild wrote:httrack crawls and downloads in rapid succession
Forked extensions :
● Add-ons Inspector ● Auto Text Link ● Copy As Plain Text ● Copy Hyperlink Text ● FireFTP button replacement ● gSearch Bar ● Navigation Bar Enhancer ● New Tab Links ● Number Tabs ● Print Preview Button and Keyboard Shortcut 2 ● Scrollbar Search Marker ● Simple Marker ● Tabs To Portfolio ● Update Alert ● Web Developer's Toolbox ● Zap Anything
Hint: If you expect a reply to your PM, allow replies...
Re: Salvaging add-ons from AMO
If you want to let it run for days on end, sure and even then, those usage patterns may get you blocked.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Salvaging add-ons from AMO
Many months ago i load whole AMO with maximum speed settings and don't get any blocked stuff.
Maybe they change that now but why should they care?
Maybe they change that now but why should they care?
Re: Salvaging add-ons from AMO
It's considered network abuse if you make an unfair amount of requests (not normal browsing behavior) and/or use up a disproportionate amount of bandwidth.
I'm surprised you don't realize this. It's common sense.
I'm surprised you don't realize this. It's common sense.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite