uBlock deletes cookies.sqlite upon list update

[Nuck-TH]

Doesn't unchecking the box for that particular filter fix the issue?

This is "ostrich sticks head into the ground" kind of "fix". Because it does nothing to underlying issue that attempt to update this list merely triggers.
We now need to look into finding minimal way to trigger issue that will help to pinpoint its reason, not to fix symptoms.

Come on people, read(and understand) previous posts before posting

[ron_1]

Well, I'm sorry, I really don't get the jist of this. If the problem is happening on my computer, I'm not seeing it. My cookies.sqlite file is apparently not being deleted because it has a creation date of Nov 29. And I am using justoff's version of uBo, but without the offending filter (it is unchecked), so I don't see why you consider that as only a band-aid to the problem. It's a non-existent filter anyway. But I guess I just don't understand.

[Bilbo47]

Doesn't unchecking the box for that particular filter fix the issue?

This ... does nothing to [the] underlying issue ... We [need to find how] to trigger [the] issue

Let me fix this for you: "Thanks for the suggestion of this diagnostic. My result of this test is XYZ."
So I agree this is not a fix, and we are still in the discovery phase, but there's no need to be snarky about it.

[Sessh]

Oh I see I am using JustOff's uBlock version not gorhills. Got confused.

Another question. Why would this issue happen only once and then never again? I've not had this happen again after the first time it happened which was likely the first manual restart of Pale Moon after the recent update. Is it just a one-off issue? I've disabled the list causing the issue for now just in case.

[Nuck-TH]

Oh I see I am using JustOff's uBlock version not gorhills. Got confused.

Another question. Why would this issue happen only once and then never again? I've not had this happen again after the first time it happened which was likely the first manual restart of Pale Moon after the recent update. Is it just a one-off issue? I've disabled the list causing the issue for now just in case.

Once database broken and deleted, then on next browser start empty new one is created. Domains there would be made in accordance to new TLD list, so the issue won't show up.

[ron_1]

This will *probably* be my last post here, since I don't understand (I freely admit), but since I would like to know, since I use justoff's uBo: isn't the problem with the add-on, and not the browser?

[Enobarbous]

This is "ostrich sticks head into the ground" kind of "fix". Because it does nothing to underlying issue that attempt to update this list merely triggers.
We now need to look into finding minimal way to trigger issue that will help to pinpoint its reason, not to fix symptoms.

This is a bit of a stupid idea, but what if you disable this filter in the malware section and manually add https://curbengh.github.io/malware-filt ... online.txt or https://malware-filter.pages. dev/urlhaus-filter-online.txt as a custom filter?
These are just mirrors of this filter, but maybe there is some problem when loading from gitlab (this is one of only 3 filters in assets.json that is loaded from gitlab, the others are taken from other sites). You can also check if this cookie problem occurs when updating the “Phishing URL Blocklist” filter (also from gitlab)
I'm digging into the uBlock filter loading/updating mechanism right now and I don't see any work with cookies at all. When working with sql there are vacuum calls, but only for /extension-data and there should be console messages in error cases...

Upd. The only calls to tld list in ublock are related to the URI.domainFromHostname function and there is work with domainCache there, but this is just Map() and as far as I understand it is just stored in memory...

[Moonchild]

Upd. The only calls to tld list in ublock are related to the URI.domainFromHostname function and there is work with domainCache there, but this is just Map() and as far as I understand it is just stored in memory...

That's good to know. The list should still be updated but we're kind of back to square 1 on figuring out what on earth is going on with the update. And it's still not clear why it broke on the eTLD list update...
In the interim I think what needs to be done is the offending list update should be disabled and a new version of uBlock pushed out to prevent people from losing their cookies constantly.

[Enobarbous]

Finally was able to replicate this error. Still can't figure out why it's the cookie.sqlite problem, but the chain looks like this: ublock 1.16.4.30 tries to update the "Online Malicious URL Blocklis" filter, can't do it (because the addresses in assets.json in one case are no longer available, in another case it returns a redirect to the gitlab login page) and, well, something happens (maybe ublock is trying to parse the returned page as a list?), resulting in sql database breaks (still don't understand why cookie.sqlite, logically ublock0.sqlite should break)
If you use uBlock 1.16.6b1 by uCyborg (which has updated addresses in assets.json) or add the actual "Online Malicious URL Blocklis" manually in "Custom" section, everything works fine.

Upd. The error also does not occur if in ublock 1.16.4.30 update assets.json to the version from 1.16.6b1, but only before installation - it seems that download addresses are also cached (or maybe it's enough to update assets.json in the installed extension and remove ublock0.sqlite along with it, then update all filters again)

The only thing unclear is why this all breaks exactly in 33.5, logically this problem should have appeared earlier...

[Enobarbous]

Off-topic:
Сон для слабых...

That's good to know. The list should still be updated but we're kind of back to square 1 on figuring out what on earth is going on with the update. And it's still not clear why it broke on the eTLD list update...

Leaving aside the previous post, I think I was able to localize the problem and it is not related to ublock.
To reproduce, in a clean PM 33.5 profile put cookies.sqlite from post viewtopic.php?f=46&t=31907&start=20#p258098 (any other cookie created in PM 33.4 and earlier should work), check that the cookies are normally visible in palemoon, then follow the link https://curben.gitlab.io/malware-filter ... online.txt
After the page is fully loaded (it redirects to https://gitlab.com/users/sign_in) I get cookies.sqlite.bak and cookies.sqlite.bak-rebuild in the profile.

[Moonchild]

Unable to reproduce on an older portable, fresh upgrade to 33.5.0 with pre-existing cookies.
Link followed, it redirects to the gitlab login page, but the cookies database is left intact, even with it writing a new permissions cookie for the site to it.

Was able to reproduce with the cookie database with pre-existing gitlab.io cookies (finally).
The issue seems to be that if cookies already exist with a gitlab.io baseDomain in the db, and it finds a redirect trying to set cookies on curben.gitlab.io/projects.gitlab.io from https headers, with the new eTLD list, this baseDomain should be curben.gitlab.io/projects.gitlab.io respectively according to the new rules. I'm not sure why this does not gracefully recover in gitlab.io's case, specifically. It seems to be fine elsewhere...

EDIT: Some research later...
Looks like this is a combination of unfortunate changes working in tandem:

1. baseDomain use was removed in Firefox, so this no longer impacts Mozilla-land
2. No.1 depends on no longer lazily loading the cookie service (which was added in Moz's never-ending chase of shorter start-up times, no matter that cookies are so ubiquitus that you always need the service anyway... Thankfully that can be followed with the change fairly close to our fork point (Firefox 58) so that should not be too difficult bug #870460
3. The public suffix list (PSL) has been morphed/redefined into something it was not really meant/designed for: it is now a way to peddle subdomain vendors' domain segregation (gitlab.io is also guilty of that). Mozilla didn't run into this issue we are having now because this wasn't common practice before they dropped the baseDomain concept.

So, at the moment, the update to the PSL clashes with something we are still using to look up cookies as it changes data out from under us, and until we stop using baseDomain as our index, we risk issues with the cookies db with a PSL update. Mozilla users also ran into our issue though, so they did get blindsided similarly.

Either way, this can be fixed by not relying on baseDomain's db entry, and simply recomputing it when reading the cookies from the db. Yes, this might incur a very small cost calling into the mTLDService on cookie read, but I doubt that will impact anyone. I'll file the relevant issues for it.

[Moonchild]

Ported several things across in Issue #2672 (UXP) and Issue #2670 (UXP) which makes this no longer break.
Further cleanup to remove baseDomain altogether Issue #2671 (UXP) needed but that's not high priority.

Merry Christmas y'all. I'm done for the year.

[suzyne]

Off-topic:
While I haven't contributed to solving this problem, kudos to the tenacity of the development team and contributors of suggestions and cases that can (sometimes?) produce the error. Really, this is interesting stuff, and I appreciate the effort at a manic time of year!

[BenFenner]

Always nice to finally find the source of such a nuanced and illusive bug. Agreed, great work all around. Nicely done.

[Sessh]

Just going to echo the sentiments. Great job everyone. I'll re-enable that list and wait for the next PM update. Enjoy the holidays everyone.