Help developing a Secret Agent alternative. Topic is solved

Add-ons for Pale Moon and other applications
General discussion, compatibility, contributed extensions, themes, plugins, and more.

Moderators: FranklinDM, Lootyhoof

Eurythrace

Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-24, 23:05

Hi,

After carefully looking at the Secret Agent add-on and trying unsuccessfully to contact the developer because of an issue it has with clearing the general.useragent.override.<domain> preferences, plus it appears to sniff and modify EVERY HTTP request, I decided to attempt to develop a PM only alternative add-on. However, this is my first foray into this area and I would appreciate some help/guidance. I have been trying to find help by searching, but resources on developing an add-on seem to be scarce and/or clear as mud.

Specifically, at the moment, I need help for the following issues:
  • 1) Is there a particular event that is fired when all tabs are closed? This would seem to be related to the browser.tabs.closeWindowWithLastTab preference. What I want is a way to tell if all active tabs/pages have been closed so only the home/start/blank page is active. I could try to keep track of what pages are opened and when they all are closed to do what I want in item 2 instead, but this seems chancy.
  • 2) If such detection is possible, then is there a way to call whatever does the privacy.clearOnShutdown preference? What I want is to clear history automatically when I detect the event in my first item. This is to as unobtrusively as possible handle ETag fingerprinting by eliminating them when it is reasonable to do so - when no pages are currently active. This would also seem to be the most reasonable time to do a random user agent change.
  • 3) A different approach would be if it is possible to eliminate all history (cache, cookies, etc.) of a specific domain when that domain is left, but I hold little hope for that. If such is possible, then I could selectively clear history when a domain is left.
I have the basic flow mapped for what I want to do, but trying to find these specific browser internal API's etc, is difficult at best. If someone is already doing this, I have no pride of authorship and will be glad to discuss my concepts and what I have done so far. In that regard, most of the random user agent changers seem to take a shotgun approach with having dozens of unique user agents that are switched out with every page load, but this is defeating the purpose of trying to blend in! Looking at desktop browser statistics, only THREE browsers would seem to be necessary: Chrome, Firefox, and one other cross-platform one like Opera or Vivaldi. Using the last 5 versions would seem to be adequate, with using Windows 7, 8.1, and 10 (32, WOW, 64), Mac 10.11.6, 10.12.1, 10.12.2, and 12.3, Linux i686, i686 on x86_64, and x86_64.

I realize there are very sophisticated methods of browser fingerprinting, but as the default general.useragent.override.<domain> preferences shows, even the big guys like Google, Facebook, etc. aren't fully beyond using the user agent string.

Thanks for any assistance in advance. Even telling me it is a waste of time with a good explanation as to why would be appreciated!

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Help developing a Secret Agent alternative.

Unread post by Moonchild » 2017-03-25, 11:56

Point 3 might actually be the easiest to implement, considering our library (history) as well as our permissions manager already have the code to "forget about a site" which clears history and per-site settings/data, respectively.

See the relevant JS in our tree in /toolkit/forgetaboutsite
This is loadable in an extension with:

Code: Select all

Components.utils.import("resource://gre/modules/ForgetAboutSite.jsm");
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-26, 01:04

Moonchild wrote:See the relevant JS in our tree in /toolkit/forgetaboutsite
Thank you so much, Moonchild. But I am a total neophyte with this stuff, so please forgive my utter ignorance. I tried to go to:
developer.palemoon.org/toolkit/forgetaboutsite
but it just gives me a 404 error. I have found this to be true of most of the developer site pages. What am I missing, please?

User avatar
Nigaikaze
Board Warrior
Board Warrior
Posts: 1322
Joined: 2014-02-02, 22:15
Location: Chicagoland

Re: Help developing a Secret Agent alternative.

Unread post by Nigaikaze » 2017-03-26, 01:37

Eurythrace wrote:What am I missing, please?
I'm pretty sure Moonchild means look at that part of the Pale Moon source code, not the developer.palemoon.org site.
Nichi nichi kore ko jitsu = Every day is a good day.

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-26, 02:46

Nigaikaze wrote:I'm pretty sure Moonchild means look at that part of the Pale Moon source code, not the developer.palemoon.org site.
Thanks so much, Nigaikaze! One less source of frustration gone, only a couple million more to go. :lol:

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-26, 11:44

OK, next simplistic question:

How do you specify a modules directory in the chrome.manifest file for a bootstrap style add-on? There seems to be contradictory info scattered around Mozilla. The best that I can find so far is that bootstrap chrome.manifest's CANNOT use the resource instruction, only the: manifest, content, locale, skin, and override instructions. If that is so, then I suppose you structure your directory as:
[content]
[modules]
module01.jsm
module02.jsm
etc.
[defaults]
prefs.js
bootstrap.js
chrome.manifest
icon.png
icon64.png
install.rdf
options.xul
In the install.rdf, you identify your addon as:

Code: Select all

<em:id>cool-toy@addons.palemoon.org</em:id>
Then the chrome.manifest file has one line:

Code: Select all

content    cool-toy    content/
Then finally in the bootstrap.js file:

Code: Select all

const { classes: Cc, interfaces: Ci, results: Cr, utils: Cu } = Components;

Cu.import( "chrome://cool-toy/content/modules/module01.jsm" );
Cu.import( "chrome://cool-toy/content/modules/module02.jsm" );
etc.
Cu.import( "resource://gre/modules/ForgetAboutSite.jsm" );
But this fails with a NS_ERROR_FILE_NOT_FOUND for all the cool-toy module imports. So what am I missing this time?

Also, although I have not got that far yet, do I access the clearing history, etc. of a domain as:

Code: Select all

ForgetAboutSite.removeDataFromDomain( domain );
?

Thanks in advance.

New Tobin Paradigm

Re: Help developing a Secret Agent alternative.

Unread post by New Tobin Paradigm » 2017-03-26, 11:51

Two items.. Do not use addons.palemoon.org or any Pale Moon domain as part of your extension ID.

Now.. to answer your question.. https://developer.mozilla.org/en-US/docs/Chrome_Registration#Manifest_Flags

Normally, you would have a top level modules directory and use a resource manifest flag to define it that way you can use a resource:// uri.

I find it easiest to just do: https://github.com/binoc-central/abprime/blob/TRUNK/src/chrome.manifest.in#L7 and just have resource://chrome-name/modules/module.jsm

As far as I know.. jsm files can't be loaded from chrome addresses.

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-26, 14:06

First off, thank you so much for the reply, Matt!
Matt A Tobin wrote:Two items.. Do not use addons.palemoon.org or any Pale Moon domain as part of your extension ID.
Got it. That was meant more as an example, but point taken.
Matt A Tobin wrote:As far as I know.. jsm files can't be loaded from chrome addresses.
This from https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules/Using
If you're writing an extension for Firefox 4 and already have a chrome.manifest with a content instruction in it, you can put the code module in your content folder and reference it like your other content files via chrome://<yourextension>/content/<yourmodule>.jsm.
Which is where I got the final directory structure I tried to use.
Matt A Tobin wrote:Now.. to answer your question.. https://developer.mozilla.org/en-US/docs/Chrome_Registration#Manifest_Flags

Normally, you would have a top level modules directory and use a resource manifest flag to define it that way you can use a resource:// uri.
I find it easiest to just do: https://github.com/binoc-central/abprime/blob/TRUNK/src/chrome.manifest.in#L7 and just have resource://chrome-name/modules/module.jsm
resource @ADDON_CHROME_NAME@ /
And the method you referred to is EXACTLY what I tried to do at first to use a resource://chrome-name/modules/module.jsm URI.

BUT!

From https://developer.mozilla.org/en-US/docs/Chrome_Registration, tucked away at the very bottom:
The following instructions are supported in Bootstrapped extensions:
manifest
content
locale
skin
override
So the resource instruction is not available in bootstrap addon chrome.manifest files, as I read it. So I am in a total quandry. It almost seems like there is no easy way to use modules in a bootstrap addon. And I tried to look through how you load modules (top level lib directory) in Adblock Latitude, but that looks extremely complex for just a couple of modules (besides going totally over my head :lol: ) Any other pointers? Please???

New Tobin Paradigm

Re: Help developing a Secret Agent alternative.

Unread post by New Tobin Paradigm » 2017-03-26, 18:34

Hmm... Guess you're right.. Maybe just make it pure toolkit. Then you have access to everything without much trouble.

EDIT: https://github.com/Pale-Moon-Addons-Team/devtools/blob/TRUNK/src/bootstrap.js.in

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-27, 01:17

Matt A Tobin wrote:Maybe just make it pure toolkit. Then you have access to everything without much trouble.
Yes, I think that going non-bootstrapped seems to be the easiest way for me. I saw the method you referenced in the Firebug addon, but it still seems overly complex for a single module. I COULD just have that module's code directly in the bootstrap.js file, but that just doesn't sit well with me. So, non-bootstrapped it is.

Thanks for everything!

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-03-28, 08:56

Just a quick FYI for anyone interested:

Defining

Code: Select all

const { classes: Cc, interfaces: Ci, results: Cr, utils: Cu } = Components;
throws a TypeError on Cc because it is first in the list. Apparently they are already defined elsewhere and left in the scope of the extension - and I HOPE they stay defined as expected! And I do the declarations first thing BEFORE importing any other modules. Must not break anything, but to get rid of the error message, I did

Code: Select all

if( typeof( Cc ) === "undefined" ) { const Cc = Components.classes; }
if( typeof( Ci ) === "undefined" ) { const Ci = Components.interfaces; }
if( typeof( Cr ) === "undefined" ) { const Cr = Components.results; }
if( typeof( Cu ) === "undefined" ) { const Cu = Components.utils; }
BTW, does anyone know of a good technique to determine if a document is from an external domain, besides do tests on the document.domain property? I ask because using a window mediator, filtering on "navigator:browser", does NOT filter file:/// or pages like Add-ons Manager, about:config, etc.

Thanks in advance.

Cheers.

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-04-01, 01:16

Hi.

I have a couple more urgent questions, please.
  • 1) I read somewhere I cannot find again, that an add-on/extension only has context within a single gBrowser object and NOT across multiple gBrowser objects. Is this correct? That is, I only need to enumerate across the tabs in my current gBrowser object and not have to invoke a Window Mediator to enumerate across all gBrowser objects? This is particularly in relation to ForgetAboutSite - when I invoke this is it only within the context of my current gBrowser, or does it span all gBrowsers?
  • 2) Am I correct in reading ForgetAboutSite, that if I pass it a root domain, it will erase ALL related subdomains too?
Thanks in advance.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Help developing a Secret Agent alternative.

Unread post by Moonchild » 2017-04-05, 23:03

1) I'm not entirely sure what you're asking. gBrowser is a DOM pointer to the <tabbrowser> XUL element (one such element exists for each browser window opened). I'm not sure what you need to enumerate -- if you overlay browser.xul with your extension, you will have one instance running per window. talking to one gBrowser and the tabs within it.
Extensions run in Chrome context. Anything you do there that manipulates browser site data will do so for the entire application. If you tell it to forget about a site, the application will forget about a site. Does that answer your question?

2) I'm pretty sure it does.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-04-06, 05:57

Thank you for your fast reply, Moonchild.
Moonchild wrote:if you overlay browser.xul with your extension, you will have one instance running per window. talking to one gBrowser and the tabs within it.
On point #1, what I was trying to ask is, when I enumerate the open tabs, I ONLY need to do so for the gBrowser of the current OS window of PM, because my current add-on instance has no context/interaction with any tabs in a different OS window of PM. Any other OS window of PM would have a separate instance of my add-on running in it. Right now I am invoking a Window Mediator which gives access to ALL gBrowser objects and I am looking at ALL open tabs in ALL OS windows of PM. But if I understand what I had read before, and what you replied above very aptly, I do NOT need to do that. All I need to do is examine all the tabs in my local gBrowser object, because there is an independent instance of my add-on running in the context of each separate gBrowser. It is obviously much simpler and faster if I only have to examine the open tabs of my local gBrowser object.

What I do is see if the root/base domain of a tab that was just closed was the last open tab that had that root domain. If it was, then I call ForgetAboutSite for that root domain. I also check to see if there are ANY open tabs to any "external" domain (not a PM about page, etc.), besides the home page (if it is "external"), and if there are not, THEN I set a new random user agent. That way, a domain does not see a new user agent while it is still open - trying to blend in, not stand out.

In that regard, do you have a module already that extracts the root/base domain from an arbitrary domain? I am doing a VERY simplistic approach of checking the TLD length. If it is TWO (2), then I check the second level domain length to see if it is THREE (3) or less. If so, then I assume the root/base domain is something like "example.nhs.uk" or "example.com.au". Otherwise, I assume it is something like "example.com". This is obviously going to fail for something like "example.parliament.uk" ( only the Brits. :lol: ), so if you already have a module that does a more rigorous check, that would be excellent.

Just a thought, have you thought about extending the general.buildID.override, general.oscpu.override, and general.platform.override preferences so that they can track the malformed domains list of general.useragent.override.<domain>? I handle this now by seeing if a new page matches any in the malformed domains list, and if so, then temporarily setting the general.buildID.override, general.oscpu.override, and general.platform.override values to match the user agent of the malformed domain from the list. I wait for half a second or so after the page has loaded to let that page do any window.navigator.* snooping it may do, then revert the general.buildID.override, general.oscpu.override, and general.platform.override values back to match my current general.useragent.override value. If you extended those preferences like the general.useragent.override.<domain> list, that would not be necessary. Just an idea for your consideration.

Cheers.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35477
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Help developing a Secret Agent alternative.

Unread post by Moonchild » 2017-04-06, 22:39

If you want to check if the last tab on a certain domain has been closed, you will need to check all OS windows if you want to be thorough. A user can have the same site open in multiple browser windows, after all.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-04-12, 05:02

Thanks for all the assistance! I am ready to submit “Eclipsed Moon” for “crush” testing. What is the next step, please? Do I attach a copy of the .xpi file here? I am looking forward to see how it fares “in the wild”.

Cheers.

ArchieLukas

Re: Help developing a Secret Agent alternative.

Unread post by ArchieLukas » 2017-04-14, 15:43

parliament.gov.uk

would be the official suffix!

Brit :mrgreen:

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-04-16, 19:43

ArchieLukas wrote:parliament.gov.uk

would be the official suffix!

Brit :mrgreen:
FYI, Wikipedia says:

Second-level domains
Active

.ac.uk: academic (tertiary education, further education colleges and research establishments (such as the British Antarctic Survey) and learned societies.
.co.uk: commercial and general
.gov.uk: government (central, devolved and local)
.judiciary.uk: courts (to be introduced in the near future)[17]
.ltd.uk: limited companies
.me.uk: general use (usually personal)
.mod.uk: Ministry of Defence and HM Forces public sites
.net.uk: ISPs and network companies (unlike .net, use is restricted to these users)
.nhs.uk: National Health Service institutions
.nic.uk: network use only (Nominet UK)
.org.uk: general use (usually for non-profit organisations)
.parliament.uk: parliamentary use (only for the UK Parliament and the Scottish Parliament)
.plc.uk: public limited companies
.police.uk: police forces[18]
.sch.uk: local education authorities, schools, primary and secondary education, community education

:lol:

Eurythrace

Re: Help developing a Secret Agent alternative.

Unread post by Eurythrace » 2017-04-21, 06:00

After playing with ForgetAboutSite, there IS a potential “gotcha” with the cookie deletion portion. As implemented:

Code: Select all

// Cookies
let cm = Cc["@mozilla.org/cookiemanager;1"].getService(Ci.nsICookieManager2);
let enumerator = cm.getCookiesFromHost(aDomain);
while (enumerator.hasMoreElements()) {
	let cookie = enumerator.getNext().QueryInterface(Ci.nsICookie);
	cm.remove(cookie.host, cookie.name, cookie.path, false);
}
But the getCookiesFromHost method will only match sub1.example.com and example.com if aDomain = sub1.example.com. IT WILL NOT MATCH sub2.sub1.example.com per here; which can be disconcerting if one is expecting cookies from ALL levels of subdomain to be deleted. I wrote a quick fix as follows:

Code: Select all

// Cookies
let cs = Services.cookies;
let enumerator = cs.enumerator;
while( enumerator.hasMoreElements() )
{
	let cookie = enumerator.getNext().QueryInterface( Ci.nsICookie2 );
	if( hasRootDomain( cookie.host, aDomain ) )
	{
		cs.remove( cookie.host, cookie.name, cookie.path, false );
	}
}
Yes, I realize that is iterating over EVERY cookie, but it seems that is the ONLY way to delete cookies from ALL subdomains of a root domain. I hope that may prove to be of use.

Cheers.

vorkot11

Re: Help developing a Secret Agent alternative.

Unread post by vorkot11 » 2017-06-15, 23:01

Eclipsed Moon 1.03 on Pale Moon 27.3.0 (32-bit) stops sending any agent string (i.e. it shows up as blank on sites and doesn't pop up when you hover over the Eclipsed Moon icon) from time to time for no apparent reason. Some sites do not work as a result. This happens both in 'Smart' and 'Static' mode. 'Smart Erase' is off. Switching it off and back on again fixes it, so is clicking "switch agent now". What could possibly be causing this?

Locked