Questions about PM extension parity, development and documentation

Discussions about the development and maturation of the platform code (UXP).
Warning: may contain highly-technical topics.

Moderators: trava90, athenian200

Public Enema

Questions about PM extension parity, development and documentation

Unread post by Public Enema » 2018-08-22, 18:24

My first post, so here's the requisite big thank you for all the effort you put into this lovely browser, a safe haven from the benighted tracking-filled, WebExt-using competition.

I'm a beginner when it comes to programming, and partly as a way to learn more, I have been researching for a little while how to edit and eventually create extensions, especially now that PM has a near monopoly on the legacy ones. I sought all the reference I could it wasn't what I expected. Unfortunately, while Mozilla's documentation for WebExt is neatly collected, the docs for legacy extensions are a planecrash on a trainwreck. It's spread over several sites, it's not properly linked or categorized and it's full of gaps. PM's docs are, well, just plain not there. So I figured I'd have to delve into PM's source code. Oh well. But I'd like to ask a few questions first.

Since the stable version 28 just hit and it seems to include a lot of changes to the JS and DOM APIs as well as to the devtools, I figured this is the perfect time to ask a few things that I haven't been able to find an answer to in these forums nor elsewhere. Excuse me beforehand if I demonstrate too much ignorance here.
  1. The most important is probably how PM's extension support matches up to FF's. While I'm well-aware that PM has changed way past the point it can be called a mere fork, this is an area where they can be compared. The problem is, I have found conflicting information. PM 27 itself identified as FF 27 to AMO so I figured that was the parity, yet I have seen posts here saying that PM can run extensions aimed up until FF 32 or even 52. I figure PM 27 or 28 extension codes aren't an exact match with those of any FF version, but if possible I'd like to know, which is the latest FF version whose full extension APIs are all included in PM 28? Or at least, the closest FF version, I suppose.
  2. It was my hope that somewhere out there, an IDE or text editor had managed to compile all l/g/d to make up for Mozilla's messy docs, but no dice. The best things I found were VSCode and WebStorm, which can use DefinitelyTyped library/grammar/dictionary (it seems everyone uses a different term) for JS, HTML, DOM and CSS as well as several Web APIs but these seem incomplete; after those there's Sublime which has all those minus Web APIs plus XUL but all in its own l/g/d format, or it can use DefinitelyTyped; and finally there's Komodo, which has all previous minus Web APIs plus XPCOM but again they're all in its own format or it too can use DefinitelyTyped. I found nothing on FUEL, Mozilla's JSM modules and most of its the many, many Web APIs. Incidentally, DefinitelyTyped also has the SDK and WebExt APIs which Debugger for Firefox uses, but these don't interest us. Unless I'm mistaken, I listed all of the APIs available to legacy extensions devs, am I correct? If not, what am I missing?
  3. Continuing from the previous item, it's important to note is that I didn't manage to get any of the 4 softwares I listed to have all code intel features I mentioned above, but then again, I am a novice. Is it possible or am I asking for too much?
  4. Regardless, it seems that, even if I got any IDE or text editor to have perfect code intel for the existing l/g/d, there would still be a need to make l/g/d for FUEL, JSM modules and most Web APIs. I presume this would require diving into PM's code. From what I (think I) learned, it would be possible to generate a skeleton documentation for all of PM's Javascript APIs by parsing the IDL and WEBIDL files and certain JSM files in order to obtain the entire PM JS l/g/d, plus whichever little information they might have in the way of commented-out text. Is this right? Also I've seen AIDL files but these seem exclusive to Android and WIDL and WIDLPROCXML which are related to tests, so these don't count, I guess.
  5. I have looked into several IDEs and expansible text editors and didn't find anything specifically aimed at FF legacy extension devs, preferably one that doesn't include code intel for FF's/PM's internal code, which extension devs can't use. My definition of code intel would be: code completion; symbol navigation ("go to definition", "list all references" etc.); indexing the users code and adding it to the workspace's l/g/d; type inference; and lastly, calltips or links to web docs. I unfortunately managed to find no IDE or text editor that could satisfy all these requirements. The aforementioned VSCode has an extension called Debugger for Firefox which supports all types of FF extensions -- overlay, bootstrapped, SDK and WebExt -- in addition to allowing debugging of web apps, but it does nothing for code intel. Is there some setup recommended for PM extension devs?
Hopefully it might be possible to cobble together some proper documentation for overlay and bootstrapped addons specifically for PM, and maybe for FF users who decide to stick with versions 56-.

Thanks for your time.

User avatar
Pallid Planetoid
Knows the dark side
Knows the dark side
Posts: 4279
Joined: 2015-10-06, 16:59
Location: Los Angeles CA USA

Re: Questions about PM extension parity, development and documentation

Unread post by Pallid Planetoid » 2018-08-22, 22:17

I know this is not what you are asking (I can't help you with the technical [coding] details regarding your questions) but here is a list of Pale Moon browser add-ons currently available (FWIW): https://addons.palemoon.org/extensions/... also many of the AMO extensions will work on Pale Moon depending upon compatibility (using the Moon Tester Tool is helpful in this regard).... I'll leave the highly technical questions for qualified others. ;)
Current Pale Moon(x86) Release | WIN10 | I5 CPU, 1.7 GHz, 6GB RAM, 500GB HD[20GB SSD]
Formerly user Pale Moon Rising - to provide context involving embedded reply threads.
Good judgment comes from experience and a lot of that comes from bad judgment. - Will Rogers
Knowing Pale Moon is indisputably #1 is defined by knowing the totality of browsers. - Pale Moon Rising

vannilla
Moon Magic practitioner
Moon Magic practitioner
Posts: 2183
Joined: 2018-05-05, 13:29

Re: Questions about PM extension parity, development and documentation

Unread post by vannilla » 2018-08-22, 23:57

I can answer a few of the technical (as in code-related) ones myself.

The version PM uses to identify itself on AMO has nothing to do with add-on compatibility itself.
Some versions catering to higher Firefox versions will work flawlessly, others not so well. It depends and testing them yourself is the only way to know.

There are no IDEs for XUL-based ("legacy") extensions. If there were any at some point in time, they have been discontinued.
But to say the truth, you don't need an IDE: a XUL-based extension is essentially XML and javascript. You don't even need HTML or CSS.
As long as you can find an editor that can automatically close XML tags even if you don't specify a schema, you're done.

About API documentation, MDN still has the XUL and XPCOM API archived. Sure, they made it harder to find them, but they're there, so if you need to look for a certain method or XUL element specification you can go there (or use a web search engine and always including "xul" as a search term).

I don't really understand the whole thing about DefinitelyTyped and all that jazz. If it's something like autocompleting stuff, I don't use any of it and never had issues.

The real issue is indeed that there aren't any beginner-friendly articles about developing an add-on.
It takes a bit of imagination and scavenging the MDN archive to figure out how to do some stuff.

Also, when working with UI (which is, like, 90% of the time), the only way to know what ID to use to position elements is to read the browser.xul file itself.
The search function on xref.palemoon.org is pretty great and makes finding stuff pretty easy.

Fedor2

Re: Questions about PM extension parity, development and documentation

Unread post by Fedor2 » 2018-08-23, 00:20

Public Enema
You have done good inquire. I myself never used such tools, no really need and no much time to test them all, but i would like to have a visual xul editor, something like office vba, in case the best i know of code edit. Did you encounter tool with this functionality?

Making superior tools only for extensions is not worth effort, so look higher. UXP very need to have good development environment with compiled help, this is important for UXP become such a thing, instead only Palemoon or Basilisk.

User avatar
ketmar
Lunatic
Lunatic
Posts: 369
Joined: 2015-07-28, 11:10
Location: Earth

Re: Questions about PM extension parity, development and documentation

Unread post by ketmar » 2018-08-23, 00:46

vannilla wrote:Also, when working with UI (which is, like, 90% of the time), the only way to know what ID to use to position elements is to read the browser.xul file itself.
not really. ;-) Element Inspector extension can inspect Chrome windows too, so you can use its picking tool to select a Chrome element and find you way from that point with a nice GUI.

vannilla
Moon Magic practitioner
Moon Magic practitioner
Posts: 2183
Joined: 2018-05-05, 13:29

Re: Questions about PM extension parity, development and documentation

Unread post by vannilla » 2018-08-23, 10:50

ketmar wrote:
vannilla wrote:Also, when working with UI (which is, like, 90% of the time), the only way to know what ID to use to position elements is to read the browser.xul file itself.
not really. ;-) Element Inspector extension can inspect Chrome windows too, so you can use its picking tool to select a Chrome element and find you way from that point with a nice GUI.
Right, I had forgotten about those kind of tools.
But isn't that just a way to read the browser.xul file? :lol:
I'm kidding, but reading the source is the only way to see how to use some not-really-documented APIs, so might as well do it for UI too.
That's what I do, at least.

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Questions about PM extension parity, development and documentation

Unread post by Isengrim » 2018-08-23, 11:30

Public Enema wrote:The most important is probably how PM's extension support matches up to FF's. [...] which is the latest FF version whose full extension APIs are all included in PM 28? Or at least, the closest FF version, I suppose.
Pale Moon is roughly equivalent with Firefox 52 in terms of extension support, sans WebExtensions. Any documentation or examples you find on overlay, bootstrap, or jetpack extension development for Firefox should mostly apply to Pale Moon 28. See also this post. There should be a number of good links in there to get you started.
Public Enema wrote:I have looked into several IDEs and expansible text editors and didn't find anything specifically aimed at FF legacy extension devs [...] Is there some setup recommended for PM extension devs?
Unfortunately, there is no complete IDE out there for XUL-specific extension or application development, or at least not one that is maintained. Your best bet is the browser's built-in devtools (element inspector, JS/error console debugger, style editor, etc.), the MDN references linked above and elsewhere, and a good-ol-fashioned text editor. If you want to use the dev tools for extension work, you'll need to enable chrome and remote debugging (see here). The dev tools do have some code completion available in the console as well as some limited cross-referencing capabilities, which may prove useful to you.
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

User avatar
ketmar
Lunatic
Lunatic
Posts: 369
Joined: 2015-07-28, 11:10
Location: Earth

Re: Questions about PM extension parity, development and documentation

Unread post by ketmar » 2018-08-23, 17:06

vannilla wrote:I'm kidding, but reading the source is the only way to see how to use some not-really-documented APIs, so might as well do it for UI too.
That's what I do, at least.
yeah, me too. and Element Inspector provides a starting point for grepping in UI files. you are usually still need to dive into XUL definitions, tho -- just to be sure that you did it right. ;-)

i once started a visual XUL designer project, but for many reasons i am unable to work on it for now. it is not abandoned, though, just put on hold. i have to catch up with new XULRunner and port my standalone XULRunner framework to the new version, and this is something i cannot do right now, due to IRL and hardware issues.

Falna
Astronaut
Astronaut
Posts: 511
Joined: 2015-08-23, 17:56
Location: UK / France

Re: Questions about PM extension parity, development and documentation

Unread post by Falna » 2018-08-24, 14:01

I don't code, but do use Notepad++, which I believe includes optional XML code color coding & syntax checking. May be helpful - no doubt there are other options out there.

Forked extensions :
● Add-ons Inspector ● Auto Text Link ● Copy As Plain Text ● Copy Hyperlink Text ● FireFTP button replacement ● gSearch Bar ● Navigation Bar Enhancer ● New Tab Links ● Number Tabs ● Print Preview Button and Keyboard Shortcut 2 ● Scrollbar Search Marker ● Simple Marker ● Tabs To Portfolio ● Update Alert ● Web Developer's Toolbox ● Zap Anything

Hint: If you expect a reply to your PM, allow replies...

Public Enema

Re: Questions about PM extension parity, development and documentation

Unread post by Public Enema » 2018-09-18, 01:35

Damn, has it been that long since I posted this? I keep postponing my reply and end up overdoing it. Sorry for those of you who asked questions. Hopefully the scope of the post will make up for it.

First, a notes that may be old news for you, but regardless. Despite being unable to compile PM, VSCode is able to debug it with the use of the PM symbols files, and using the Debugger for Firefox, it can debug seemingly any FF/PM extension.

Second, should I message Moonchild to ask for a pointer on which files contain the JS code available to add-ons? I don't know how busy he is or how receptive to requests. I'm fairly sure IDL files point towards all JS APIs written in C++, but I don't know if there are other APIs written in Python or JS itself.
Pale Moon Rising wrote:Moon Tester Tool
Oh I didn't know it could help with add-on devving, thank you.
vannilla wrote:It depends and testing them yourself is the only way to know.
Ah well. Maybe if I can put together a comprehensive doc of all PM add-on entities, it would be possible to program a utility to scan a FF add-on to see if it matches PM's specifications?
vannilla wrote:But to say the truth, you don't need an IDE: a XUL-based extension is essentially XML and javascript. You don't even need HTML or CSS.
Yeah, it's just that having the reference docs on hand is just the way I prefer to code. You know how sometimes you end up coding some horribly hacky function to do something important, only to find out later that the default l/g/d already had the perfect solution tucked away in the docs?
vannilla wrote:I don't really understand the whole thing about DefinitelyTyped and all that jazz. If it's something like autocompleting stuff, I don't use any of it and never had issues.
It's the closest thing there is to standardized code intel so far. It's one or more TypeScript files usually ending in *.d.ts that lists the types, standard functions, syntax etc. of a language, tho it can be made for a 3rd party library, or some dialect, or a specific program's l/g/d such as Mozilla's add-on development. So instead of each IDE having to implement each language at a time and all by itself, they just have to make their code intel engine read those files. It's a Microsoft thing so I guess it's part of their "embrace, extend, extinguish" thing.
vannilla wrote:Also, when working with UI (which is, like, 90% of the time), the only way to know what ID to use to position elements is to read the browser.xul file itself.
I found some interesting XUL tools I describe below.
Fedor2 wrote:I myself never used such tools, no really need and no much time to test them all, but i would like to have a visual xul editor, something like office vba, in case the best i know of code edit. Did you encounter tool with this functionality?
I searched all over and did a lot of internet archeology (thanks, Archive.org), and I found a surprising amount of XUL tools. It seems that early on, FF devs talked about making a proper IDE for XUL, but they didn't get organized so intead everyone made their own tools, simpler than IDE. The downside is that most of these tools are really old for internet standards. Then again, XUL's specifications weren't altered too often, so they have a good chance of working. The tools add-ons in the next paragraph all claim to work for any version from 4 or less all the way to 56, so odds are they work fully for PM. I haven't tested most of them yet, mind you.

First, I have not tested the vast majority of the tools I'm about to list due to simple lack of time, as there's too many. That said, despite the stated above, I did find interesting tools:
  • Spket IDE: Unlike so many of the other tools here, this is a finished product. It has good code intel for XUL and other XML flavors, as well as JavaScript and some 3rd party libraries, so it can be used for developing whole add-ons. The big downside: it doesn't seem to have a visual preview. Alas. Anyway, the standalone version only works on 32-bit Windows and Linux, sadly, but it's also available as an Eclipse plugin. You can go to File > New > Project... > Spket > Firefox Extension Project to get started, and the site's support page has a link to several tutorials.
  • XUL Explorer: a standalone visual XUL editor. It has XUL reference and a visual preview of your XUL code, but it's a separate pane so you can't see the code and the preview at the same time. Also no drag-and-drop.
  • Mozdev Project Wizard: only now I discovered Mozdev has a little online tool to kickstart some Mozilla stuff, including add-ons. But it's aimed at the old FF system, which needed signing and some work to make the XPI and such.
  • XulApp: a really damn old (the dev talks about FF 1.1), unfinished IDE, but which claims to have a drag-and-drop form editor. No JS support, however. He also made library called XaLib which can be used in other projects and supposedly adds some niceties to add-on coding, including a "really simple way to access widgets from JS. Goodbye to getElementById".
  • The version in the CVS repo folder "xulapp/" is newer and has no installer, whereas the version available for download in the site, which is 0.2.13, is the one present in the repo folder "xulappold/". However, I haven't managed to get it to work at all so far, altho I think I will manage to do it.
  • XUL Gear: a XUL editor with preview window, all elements there to be inserted with a click, with attributes to be easily set and all, but it doesn't have graphical drag-and-drop, tho the treelist has it. Oh, and it has been broken since FF 20, when E4X was removed, and its reference is obviously outdated, as with most tools in this post. It seems that it can be updated to work with PM (and FF 56, for that matter) with a bit of work, or with XULRunner. In the meantime, there are two possibilities to use this tool now...
    1. The first method is slightly more inconvenient but amusingly oblique. You see, there used to be an audio player built on XUL called Songbird, which folded in 2013. It had a surprising amount of add-ons, some of which can still be rescued via Archive.org and possibly be of some use for someone. In any case, an open source, fan-made fork called Nightingale kept on going... for six more months. But at least they have hept the site and git repos going. And what's interesting is that the latest XUL Gear can be run on it with minimal alteration. Grab the latest version on the AMO link a couple of paragraphs above, open its install.rdf and replace songbird@songbirdnest.com with nightingale@getnightingale.com. There you have it, just repack it, install it in Nightingale and it works identically as it would in old FF versions. There are just three caveats: like I mentioned above, its reference is outdated (it's from FF 4.0!); the navigation toolbar button is a thin, image-less button because the dev used a SVG for a pic (but it can also be accessed via Web Developer menu); and sometimes the arrowscrollboxes in the middle section disappear (always happens when I change to layout B then to layout A). Nightingale also has its own developer tools based on the Extension Developer's Extension which I showcase a few paragraphs below, in case you have any ideas in mind. It includes XPCOMViewer, XUL Periodic Table, XUL Editor XUL Explorer and more.
    2. The second method is probably for us. Simply download FF Portable 19.0.2 from PortableApps.com's Sourceforge archive (Windows only, I'm afraid) and you can just install XUL Gear and go at it. I guess the portable FF has version compatibility check disabled. Regardless, this has the same two caveats as above -- outdated references, image-less toolbar button and buggy arrowscrollboxes in the middle section -- but has a small new benefit, in that the middle section can be scrolled with the mousewheel. Otherwise, this seems to be identical to using this tool on Nightingale or FF 4.0.
A small interruption for a question: do you think it's worth the time and effort to fix XUL Gear for PM and/or XULRunner? The former is more immediately useful to us, while the latter is easier to port and can be a "standalone" program -- which doesn't have much point nowadays, what with XUL being dead. Either way, XUL Gear could be brought up to date and used in a hypothetical PM add-on dev kit. On the other hand, I don't know if this is the best tool, so I'd rather at least someone here gave these a whirl a vote for the best XUL editor.

One of MDN's page about setting up an IDE specifically for add-on development (there's a second one, because Mozilla's docs are a mess) suggested these:
  • Komodo IDE: the centerpiece, obviously. It still uses XUL, and it's the only IDE I found with at least partial code intel for XUL, as well as Mozillas's CSS extensions.
  • DOM Inspector (add-on): apparently it can edit too.
    Venkman debugger: yeah, the MDN page in question is old, even if it was updated sometimes. This is beyond obsolete.
  • Tamper Data (add-on): not related to XUL but MDN recommended this for fooling around with HTTP headers, parameters, requests etc.
    Firebug + Chromebug: I'll expound on them below.
  • Leak Monitor: also unrelated to XUL, but potentially valuable.
The second MDN page about setting up an IDE has a long list of add-ons to help development, it's also a bit outdated. Some I mentioned above, some are beyond the scope of XUL (but I recommend you check the list regardless), so I'll list the more useful ones for us. Unlike those above, these have varying FF version requirements: Lastly, a note on ol' Firebug:
Firebug + Chromebug (Chromebug user guide: a special case. I'm fairly sure that Firebug's features have been superceded by both FF's and PM's native devtools, but some of its add-ons have specialized functions. Chromebug, specifically, lets you debug XUL applications, which is pretty much synonym with legacy addons. The problem is, the newest version of Chromebug is fairly old, and it's only compatible with a similarly old version of Firebug. Namely, the version in question is 1.8.0a2. Anyway, PM already has an option to debug restartless extensions, but as far as I know, it can't do it to overlay extensions. And God knows about PMkit ones.

And I still haven't gone through all of them, but I'm finishing the list here for now, or else I'd never finish this post. I still have to go through most of this list in the Mozilla wiki, this one relic called Vixen (which seems to have nothing to see, actually), XML editors which might happen to support XUL, some increasingly obscure candidates (one of which, ATF for Eclipse, I mention below), and I bumped onto several XUL-related projects in a Perl site. I don't know a lick of Perl, but I guess I can look around.
Fedor2 wrote:UXP very need to have good development environment with compiled help, this is important for UXP become such a thing, instead only Palemoon or Basilisk.
I'm afraid I'm still too unexperienced for helping with UXP.
Isengrim wrote:Pale Moon is roughly equivalent with Firefox 52 in terms of extension support, sans WebExtensions.
Well I'll be damned, I and a lot of other users have been missing out on a lot of newer extensions. Why not set the FF useragent in PM to 52 when visiting AMO?
Isengrim wrote:Your best bet is the browser's built-in devtools (element inspector, JS/error console debugger, style editor, etc.),
The current tools really are terrific, it's just lacking the ability to debug overlay extensions which require rebooting.
Isengrim wrote:If you want to use the dev tools for extension work, you'll need to enable chrome and remote debugging (see here). The dev tools do have some code completion available in the console as well as some limited cross-referencing capabilities, which may prove useful to you.
Huh, remote debugging is how I had been debugging in VSCode + Debugger For Firefox, but it didn't occur to me to use the native tools that way. Thanks. But that can't isolate overlay add-ons like PM's own debug button for bootstrapped add-ons can, can it?
ketmar wrote:i once started a visual XUL designer project, but for many reasons i am unable to work on it for now. it is not abandoned, though, just put on hold. i have to catch up with new XULRunner and port my standalone XULRunner framework to the new version, and this is something i cannot do right now, due to IRL and hardware issues.
Wait, new XULRunner? Mozilla stopped in 2015 and PM's hasn't been updated since 2016.

Anyway, I found something you may find useful. It seems an (abandoned) Eclipse project called ATF has managed to stuff XULRunner into a plugin, which might also actually include some useful information about FF's APIs for my reference docs.

User avatar
ketmar
Lunatic
Lunatic
Posts: 369
Joined: 2015-07-28, 11:10
Location: Earth

Re: Questions about PM extension parity, development and documentation

Unread post by ketmar » 2018-09-18, 07:54

>Wait, new XULRunner? Mozilla stopped in 2015 and PM's hasn't been updated since
>2016.

it was always possible to build XULRunner from Pale Moon sources. and it becomes an official tool for UXP, 'cause UXP is not about Pale Moon, it is about XUL applications. i am building Pale Moon from sources anyway, and so i did with XULRunner, which i used for several my apps and ports (including LunarIRC, which i never bothered to announce, and now it is too late, lol).

Locked