A (potential?) solution for SDK extensions...

Talk about code development, features, specific bugs, enhancements, patches, and similar things.
Forum rules
Please keep everything here strictly on-topic.
This board is meant for Pale Moon source code development related subjects only like code snippets, patches, specific bugs, git, the repositories, etc.

This is not for tech support! Please do not post tech support questions in the "Development" board!
Please make sure not to use this board for support questions. Please post issues with specific websites, extensions, etc. in the relevant boards for those topics.

Please keep things on-topic as this forum will be used for reference for Pale Moon development. Expect topics that aren't relevant as such to be moved or deleted.
dark_moon

Re: A (potential?) solution for SDK extensions...

Unread post by dark_moon » 2016-12-25, 19:16

PalemoonOriginal
---------------------------
SDC: 2095ms Startup || 812ms Loader || 512ms Bootstrap
RES: 1264ms Startup || 608ms Loader || 397ms Bootstrap
DEC: 651ms Startup || 414ms Loader || 284ms Bootstrap


PalemoonImproved
----------------------
SDC: 738ms Startup || 271ms Loader || 284ms Bootstrap
RES: 455ms Startup || 206ms Loader || 240ms Bootstrap
DEC: 248ms Startup || 148ms Loader || 199ms Bootstrap


PalemoonImproved2
-----------------------
SDC: 391ms Startup || 139ms Loader || 208ms Bootstrap
RES: 244ms Startup || 107ms Loader || 189ms Bootstrap
DEC: 139ms Startup || 81ms Loader || 172ms Bootstrap

kizo07

Re: A (potential?) solution for SDK extensions...

Unread post by kizo07 » 2016-12-25, 20:28

PalemoonOriginal
---------------------------
SDC: 2064ms Startup || 791ms Loader || 528ms Bootstrap
RES: 1250ms Startup || 586ms Loader || 416ms Bootstrap
DEC: 659ms Startup || 422ms Loader || 302ms Bootstrap


PalemoonImproved
----------------------
SDC: 770ms Startup || 274ms Loader || 326ms Bootstrap
RES: 485ms Startup || 213ms Loader || 277ms Bootstrap
DEC: 268ms Startup || 160ms Loader || 231ms Bootstrap


PalemoonImproved2
-----------------------
SDC: 407ms Startup || 142ms Loader || 248ms Bootstrap
RES: 259ms Startup || 109ms Loader || 228ms Bootstrap
DEC: 152ms Startup || 84ms Loader || 212ms Bootstrap
This is with Win fw enabled and without external sec. I tested with external sec on too, but since difference in 'Improved2' is negligible (even some ms better with sec on?!!), it's no point to publish. I tested as admin on Win 10 x64.
Just to mention, got 'Imroved' triggered in carantene, once (probably I forgot to close previous ver)
It's nothing more to do I think, just intend to send 'Imroved2' to CERN, that they see what is a Realy Accelerating Science :)

kizo07

Re: A (potential?) solution for SDK extensions...

Unread post by kizo07 » 2016-12-26, 13:50

kizo07 wrote:got 'Imroved' triggered in carantene, once (probably I forgot to close previous ver)
Umh...I saw this now. JustOff, you probably thought 'What The...Da Vinci Code Is This Now!' or 'He got some bad weed yesterday'. Real sentence is 'I got 'Imroved' triggered in quarantine, just once (I forgot to close previous section)'
...For everybody else...It's obvious. :) 'carantene' is is my own word, mixed between Norwegian 'karantene' and English 'quarantine' ;)

Once again, well done Justoff!

ThirdTryCharm

Re: A (potential?) solution for SDK extensions...

Unread post by ThirdTryCharm » 2016-12-26, 15:22

JustOff wrote:
dark_moon wrote:Did i need now to run PalemoonImproved2.cmd
My most favorite add-on is SelfDestructingCookies and VERY happy to see this thread -- BUT -- how can I apply this to the Linux version?

Thanks for the great work you have done with this, but having just left windows for the security and no-spying of Ubuntu, I seem to be out in the cold. :)

{edit}
OK, cancel that request, I have just installed Crush those Cookies from the the Palemoon add-ons. Seems to be almost as good.
{/edit}

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2016-12-26, 16:38

ThirdTryCharm wrote:My most favorite add-on is SelfDestructingCookies and VERY happy to see this thread -- BUT -- how can I apply this to the Linux version?
I hope that Self-Destructing Cookies will be officially supported starting from Pale Moon 27.1, but even now there are at least two good alternatives.

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2016-12-28, 18:01

I started to do some sketches for future PMkit documentation and found that my initial idea of renaming package.json just to avoid current ban in Addon Manager was very stupid. Let me explain why and propose more appropriate solution.

There are two tools for creating SDK-based extensions: CFX (old) and JPM (modern). The main difference between code that is produced by these tools is the way to load and unload extensions. CFX injects its own loader into every add-on, while JPM uses the loader from SDK that is embedded in browser. It is also known that CFX stores the manifest of extension in the file harness-options.json and JPM in the file package.json.

Pale Moon 26 and earlier had no embedded SDK loader required to run extensions made using JPM and only extensions that created using CFX were supported. Thus, currently there is not a single add-on made using JPM that is targeting to Pale Moon. At the same time and for the same reason there is not a single add-on made using JPM that is targeting to Firefox version less than 38.

Therefore, there is absolutely no need to ban package.json in Addon Manager, but if we want even more confidence, we can check that extension that contains this file directly targets to Pale Moon.

Code: Select all

-    addon.jetsdk = aZipReader.hasEntry("package.json")
-                || aZipReader.hasEntry("harness-options.json");

+    if (aZipReader.hasEntry("harness-options.json")) {
+      addon.mozsdk = true;
+    } else if (aZipReader.hasEntry("package.json")) {
+      let app = addon.matchingTargetApplication;
+      if (app && app.id == Services.appinfo.ID) {
+        addon.mozsdk = false;
+      } else {
+        addon.mozsdk = true;
+      }
+    }
Why it's so important to keep the name package.json? This is because it's used in JPM which is not only tool to package XPI, but also to build, run and test the extension. We would look just silly if we will tell to developers: "Install JPM and Firefox as development environment, then rename package.json and change path in bootstrap.js, repack XPI and try to run it in Pale Moon. But if something goes wrong, switch back to JPM+Firefox to debug".

As for harness-options.json, I think it should stay banned completely. Although Firefox still allows to run CFX-packaged extensions, this often leads to memory leaks because of old loader code. Given that any add-on can be easily converted from CFX to JPM there is no sense to support such compatibility in Pale Moon.

GreenGeek

Re: A (potential?) solution for SDK extensions...

Unread post by GreenGeek » 2016-12-28, 19:44

By banned do you mean they need to be blocked until modified ("hacked" ;) ) to work with PM, as you explained earlier? Because, both SDC and Decentraleyes have harness-options.json.

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2016-12-28, 20:02

GreenGeek wrote:By banned do you mean they need to be blocked until modified ("hacked" ;) ) to work with PM, as you explained earlier?
If (I hope when) PMkit will be supported officially we will not need in any hacks anymore, because the most of SDK-based addons could be adapted to PMkit using simple and documented algorithm.
Because, both SDC and Decentraleyes have harness-options.json.
The latest versions of both SDC and Decentraleyes are packaged using JPM and have no harness-options.json inside, however it doesn't matter (see post above).

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35636
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: A (potential?) solution for SDK extensions...

Unread post by Moonchild » 2016-12-29, 10:08

Your alternative proposal will cause confusion because resource paths and front-end assumptions inside the extension would be incorrect. From an add-on extension developer's point of view, what do you expect them to use anyway? JPM+Firefox would be the only option for that because we don't supply development and packaging tools for these kinds of extensions, ourselves, and JPM from Firefox doesn't know about Pale Moon as a target application either (our jetpack modules have explicit additions for Pale Moon as an application and front-end in a few places).
If you're going to have to go in to change the paths anyway, then you should be explicit and change the manifest name while you're at it.
Do you really expect SDK extensions targeting FF38 and its front-end to be compatible with Pale Moon's application code? Because you seem to assume that they are, while those would be targeting Australis... Those need to be adjusted anyway.

I don't intend for Pale Moon to come with extension SDK development tools (and somehow provide support for them with our non-existent SDK support team). If people want to use this, then they will have to take the extra steps.
Why? because of the very real risk that users (in many cases not developers) will otherwise hack install.rdf, add a Pale Moon ID as target application (which I'm sure a number already have since v25) and then expect it to "just work" while it can't (in many cases) -- and then load support for that off on us, because "it works for this other add-on".
No, on the browser side it must be distinct and different from Firefox SDK to avoid any sort of confusion or muddling of the waters. JPM may be used on Firefox to create a base SDK extension to work from, but any conversion to PMkit will always have to be done manually or it will cause problems, and JPM can't be used to run and/or debug PMkit extensions, as far as I understood.

I know the SDK caters to lazy extension development to begin with, but I'd rather not feed more into that and letting developers assume we're "just another Firefox" to them when they need to take great care if they want to adjust their work for use on Pale Moon.

Unless I'm missing something obvious here -- in that case please explain.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2016-12-29, 16:27

Moonchild wrote:Your alternative proposal will cause confusion because resource paths and front-end assumptions inside the extension would be incorrect. From an add-on extension developer's point of view, what do you expect them to use anyway? JPM+Firefox would be the only option for that because we don't supply development and packaging tools for these kinds of extensions, ourselves, and JPM from Firefox doesn't know about Pale Moon as a target application either (our jetpack modules have explicit additions for Pale Moon as an application and front-end in a few places).
JPM does not care about the inside design of jetpack modules or any specific features of target applications. It can successfully test, run and package extensions for Pale Moon 27 (the same way that it does it for Firefox, Thunderbird, SeaMonkey, etc) without need of any adjustments in its code.

JPM + Pale Moon is enough for the full development cycle. I checked it.
If you're going to have to go in to change the paths anyway, then you should be explicit and change the manifest name while you're at it.
Do you really expect SDK extensions targeting FF38 and its front-end to be compatible with Pale Moon's application code? Because you seem to assume that they are, while those would be targeting Australis... Those need to be adjusted anyway.
You are right, but if we rename package.json the developers will need to do many unnecessary and unnatural steps just to be able to use JPM.
I don't intend for Pale Moon to come with extension SDK development tools (and somehow provide support for them with our non-existent SDK support team). If people want to use this, then they will have to take the extra steps.
Why? because of the very real risk that users (in many cases not developers) will otherwise hack install.rdf, add a Pale Moon ID as target application (which I'm sure a number already have since v25) and then expect it to "just work" while it can't (in many cases) -- and then load support for that off on us, because "it works for this other add-on".
How is this different from the situation when users will rename package.json and then expect it to "just work" while it can't (in many cases) and then load support because "it works for this other add-on"?
No, on the browser side it must be distinct and different from Firefox SDK to avoid any sort of confusion or muddling of the waters. JPM may be used on Firefox to create a base SDK extension to work from, but any conversion to PMkit will always have to be done manually or it will cause problems, and JPM can't be used to run and/or debug PMkit extensions, as far as I understood.
Fortunately it can be used, and it's cool! Even when Firefox will deprecate itself by removing support of XUL and SDK add-ons we will remain solid.
I know the SDK caters to lazy extension development to begin with, but I'd rather not feed more into that and letting developers assume we're "just another Firefox" to them when they need to take great care if they want to adjust their work for use on Pale Moon.
I'm going to write the guide that will explain these differences and how to properly adjust extensions created using SDK for Firefox to run with PMkit. And I think that possibility of using JPM is important, otherwise it will be the guide how to hack instead of how to develop.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35636
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: A (potential?) solution for SDK extensions...

Unread post by Moonchild » 2016-12-29, 18:39

All right. Thanks for addressing those points.

So, if I understand correctly, JPM add-ons are even more limited and abstract in that they only supply metadata and letting the actual jetpack-embedded code handle the rest in terms of how things are loaded/initialized? (I'm deferring to you here since I have never in my life looked at the sdk-development procedure for Firefox)

In that case allowing package.json would indeed be the way to go.

As for "editing install.rdf" and "renaming package.json" as a second step is concerned, unfortunately it's not entirely what you think. as said before, people have already gotten used to editing install.rdf files because of Pale Moon's GUID.
Ultimately though, if JPM + Pale Moon can be used as-is when keeping package.json, then keeping compatibility there is good, and you should offer a PR for the add-on manager change to make this possible.

In early January, I'll make an unstable channel available for people, which will be based on our development trunk (a "nightly" of sorts but builds won't be every night) so people can have the latest builds from source to work with re: extensions.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2016-12-29, 20:24

Moonchild wrote:So, if I understand correctly, JPM add-ons are even more limited and abstract in that they only supply metadata and letting the actual jetpack-embedded code handle the rest in terms of how things are loaded/initialized? (I'm deferring to you here since I have never in my life looked at the sdk-development procedure for Firefox)
Yes, the minimal add-on source for JPM consists of two files: index.js (actual code that uses Jetpack API) and package.json (manifest). JPM uses data from this manifest to automatically create install.rdf and bootstrap.js with minimal code that passes control to Jetpack loader embedded in browser. Then loader uses that manifest file to setup proper running environment and calls the actual code of add-on.
In that case allowing package.json would indeed be the way to go.
This is exactly what I meant. The main benefit of JPM is the opportunity to write and run tests for add-on as well as debug it within separate browser profile directly from the command line. But this works only when JPM and Jetpack are using the common name of manifest file.
Ultimately though, if JPM + Pale Moon can be used as-is when keeping package.json, then keeping compatibility there is good, and you should offer a PR for the add-on manager change to make this possible.
I'll do that, thanks.
In early January, I'll make an unstable channel available for people, which will be based on our development trunk (a "nightly" of sorts but builds won't be every night) so people can have the latest builds from source to work with re: extensions.
Thanks, I think it will be very useful.

PS: I understand your concerns about users that misunderstand the nature of Pale Moon, how it different from Firefox and often abuse the support after wrong actions caused by the wrong assumptions. But in this case I see no other way to do what we want to be done well and I hope that the proper PMkit documentation will minimize the unavoidable evil.

Fedor2

Re: A (potential?) solution for SDK extensions...

Unread post by Fedor2 » 2016-12-30, 00:51

Adding jpm support is good, and I am willing to help, had some business with the matter, only when 26 pale moon has been released.


nico239

Re: A (potential?) solution for SDK extensions...

Unread post by nico239 » 2017-01-05, 14:43

I was looking for a solution for
Awesome Screenshot
Self destructing cookies

I see a solution here but how to do...

If I open PalemoonImproved2 everything's ok for surfing and Self destructing but NOT for Awesome screenshot

And the default Palemoon browser is always the blue one and not the red one (cmd)

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2017-01-05, 16:57

nico239 wrote:I was looking for a solution for
This is dev forum, you have to wait for official Pale Moon 27.1 (beta - to try, release - to use).

Today I've cooked the PMkit shim for 'sdk/ui/button'. While in-depth testing is still required, at first sight it works well both alone and in combination with panel, and probably even has no memory leaks)) If everything will be ok, this shim will allow greatly simplify the adaptation of SDK-based add-ons to PMkit.

nico239

Re: A (potential?) solution for SDK extensions...

Unread post by nico239 » 2017-01-05, 21:18

Ok thank you... I'm waiting for it

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2017-01-07, 13:32

With the latest PMkit updates I can successfully run Awesome Screenshot, Ghostery, Privacy Badger and all add-ons that previously required hacking literally untouched. They look fully functional (with all buttons and context menus) and correctly release resources on disable or uninstall.

I could also write the shims for 'ui/sidebar', 'ui/toolbar' and 'ui/frame' but I can't find a single extension that actually uses it. Correct me if I am wrong.
Moonchild wrote:Do you really expect SDK extensions targeting FF38 and its front-end to be compatible with Pale Moon's application code? Because you seem to assume that they are, while those would be targeting Australis... Those need to be adjusted anyway.
It even seems that Pale Moon 27.1 + PMkit is more compatible with Mozilla Add-on SDK than Pale Moon 24.x-26.5 was. Moreover, it turns out that SDK-based add-ons for Firefox are more compatible with Pale Moon 27.1 than classic XUL and bootstrapped ones. This is because PMkit transparently supports both 'widget' and 'ui/buttons' API, while SDK-based add-ons should not care is it Australis or classic UI as they are never working with it directly.

Thus, SDK-based extensions that target to FF38+ can run in Pale Moon 27.1 just with only change in path to PMkit, and old add-ons that were build with cfx can be converted to jpm almost automatically.

In view of the above, I would propose to take the next logical step: return the path to PMkit to its original location: 'resource://gre/modules/commonjs/'. This will simplify the jpm usage (autogeneration of bootstrap.js) and will give a good sign to developers that they are welcome to attend here in contrast with Mozilla which throws them out. If we will use the common path (as it was in Pale Moon 24.x-26.5) developers will be able to smoothly adapt their add-ons to Pale Moon and new users will be able to find their usual extensions more easily.

Relicbreaker

Re: A (potential?) solution for SDK extensions...

Unread post by Relicbreaker » 2017-01-08, 03:39

Hello there, longtime lurker, first-time poster.

I'd like to help if at all possible; specifically, I've wanted to get Video Download Helper working and back on its feet, and have been trying to do so on my own for some time, unsuccessfully I might add... :mrgreen:
This solution sounds quite elegant, and a great deal better then people like myself attempting to full-nelson our favourite extensions into submission.
So, I'd like to help if at all possible; I'm a shiterrible coder, but I'm pretty fastidious about documenting things, blame several years of being a glorified secretary...

I do not know if that'd help in the slightest, and it might be a circumstance where too many cooks spoil the broth, but in the interest of showing that not all end-users want to complain and rant all day, I'd figure I could at least offer something I know I can do. :D
Just let me know if you think you'd need a stenographer. I'm used to working with code, and cleaning it up to be viewed by other people, for whatever that's worth.
Even if that is of no use to you, thanks MoonChild for (as always) keeping the dream alive, and JustOff for your initiative and work.

JustOff

Re: A (potential?) solution for SDK extensions...

Unread post by JustOff » 2017-01-08, 13:25

Relicbreaker wrote:I'd like to help if at all possible; specifically, I've wanted to get Video Download Helper working
Thanks for your support and intention to help. While I don't quite understand how it could be used, I'm grateful for the offer 8-)

As for Video DownloadHelper, I successfully ran it in Pale Moon 27.1 + PMkit after minimal adjustments. The precise checking is still required, but at least I haven't had any problems downloading from YouTube. We must also consider that VDH is free but not open source, so Pale Moon support should be added by the author of VDH. Let's return to this subject after Moonchild will publish the official Pale Moon 27.1 beta.

Locked