uBlock deletes cookies.sqlite upon list update

[Nuck-TH]

Beats me why, but reverting c0b74ce133 in release tip indeed mitigates the issue...
Since this is bafflingly unrelated change, this isn't proper fix, but tracing what ublock does when updating filterlists and when there issue occurs exactly is beyond my skills.
Maybe due to changes in domains(i actually didn't look into what that file is and what it is for) something stops matching old cookies database and machinery working with it starts to think that it is damaged, so it (unsucessfully) tries to repair it, deleting "broken" file as part of the process. In this case it is actual bug in platform, because it both fails to repair database or even create a new one.

Would be nice if someone re-validated my experiments tho(all test builds are in ftp dir linked in earlier post).
---
Also question - can i use profile on which i downgraded to 33.4.1 upgrading back to 33.5? Or something could have been broken?

P.S. Its long overdue to change topic title, because it is a misnomer at this point.

[billmcct]

What are "cookies.sqlite-shm and cookies.sqlite-wal" for in PM?

[Nuck-TH]

What are "cookies.sqlite-shm and cookies.sqlite-wal" for in PM?

https://www.sqlite.org/tempfiles.html
First link in google...

[billmcct]

Thanks.

[Moonchild]

Maybe due to changes in domains(i actually didn't look into what that file is and what it is for) something stops matching old cookies database and machinery working with it starts to think that it is damaged, so it (unsucessfully) tries to repair it

Nice theory but if that was the case then it would happen all the time, dependent on which domains you visit. And we're not seeing that -- instead it's tied to use of an extension. So I'm really not convinced it's a platform problem; after all we've had this mechanism in place for a very long time and all I did was update the data file used by this mechanism with an updated version.

What the update does is provide a updated list of "effective top-level domains" which is important because the powers that be have been in a rather annoying habit of making 'classes" of second level domains that are treated as to-level in practice. Think .co.uk for British commercial sites. For the browser to understand that .co.uk is effectively to be treated as a top-level domain, it needs to keep that in a list, so it can treat x.co.uk and y.co.uk as different domains, not just different origins under a single domain -- which impacts things like CORS, the same-origin policy, etc.
Having an at least somewhat up-to-date list is important as it impacts some aspects of security, as these conventions do change from time to time.

Please do verify this behavior. I'd prefer not to back out this update. I'm also still not sure why uBlock wants to read/write cookies of sites as that isn't(?) part of its functionality. Not being able to recover from it may be because the extension keeps "messing it up" while it's running in the browser, and not necessarily an issue in the platform.

can i use profile on which i downgraded to 33.4.1 upgrading back to 33.5? Or something could have been broken?

Well if there has been weirdness with the profile something could have been broken. So there's some unknown issue there but there hjasn't been any data migration between the two versions so there isn't necessarily a hard "no can do" in that respect.

[RealityRipple]

There are some weird ones that get classified as effective tlds, though... CDNs and some google service URLs. It could be that some entry in the extension is being considered all-tld and thus host-less, and is matching too many entries?

[Nuck-TH]

Nice theory but if that was the case then it would happen all the time, dependent on which domains you visit. And we're not seeing that -- instead it's tied to use of an extension. So I'm really not convinced it's a platform problem; after all we've had this mechanism in place for a very long time and all I did was update the data file used by this mechanism with an updated version.

That the thing - uBlock pretty much has fixed domains list during filter lists update.
Also note that it happens only whed database is migrated from previous version(which had old tld list).
Maybe it can happen without ublock, but it would be pretty much random chance, because we don't know exact trigger conditions. Some people haven't issue even with ublock, so it should be possible to disable all filter lists and find which one(s) trigger issue.

Well if there has been weirdness with the profile something could have been broken. So there's some unknown issue there but there hjasn't been any data migration between the two versions so there isn't necessarily a hard "no can do" in that respect.

No, i did not use that profile for testing, it is my main one - i downgraded to be able to browse and not have cookies lost.
Bisect testing was done on clean portable.
I ask specifically if there was any profile related changes between this two versions. If no, so i assume profile can be used back and forth without any issues.

For the browser to understand that .co.uk is effectively to be treated as a top-level domain, it needs to keep that in a list, so it can treat x.co.uk and y.co.uk as different domains, not just different origins under a single domain -- which impacts things like CORS, the same-origin policy, etc.

Can't this be the source of issue? For example if some domain was top level and was removed or added and suddenly cookies that beloged to single tld started to belong to different or vice versa. It probably can trow off cookies machinery and make it mark database as broken.

[Moonchild]

Can't this be the source of issue?

Maybe? The cookies database does store a "base domain" in its table, but the most that would ever cause would be cookies from a now-changed eTLD no longer being recognized as belonging to that domain. It certainly wouldn't cause db corruption.

[Enobarbous]

Can't this be the source of issue? For example if some domain was top level and was removed or added and suddenly cookies that beloged to single tld started to belong to different or vice versa. It probably can trow off cookies machinery and make it mark database as broken.

Could it be because within uBlock itself, there is also an effective_tld_names.dat within the \assets\thirdparties\publicsuffix.org\list, and it's not the same as the one used in PM?

[BenFenner]

Off-topic:
.co.uk being a TLD really threw me for a loop about two years ago and turned a simple line of code I was writing into an obnoxious line of code that relied on a 3rd party library instead.

[Nuck-TH]

Out of default filters updating only one triggers the issue:
Malware domains > Online Malicious URL Blocklist

[back2themoon]

That's broken in JustOff's uBO version (dead link).

[Moonchild]

Could it be because within uBlock itself, there is also an effective_tld_names.dat within the \assets\thirdparties\publicsuffix.org\list, and it's not the same as the one used in PM?

bingo.

Not sure why it doesn't rely on the machinery inside the browser for getting the eTLD. I'm pretty sure that can be accessed through an interface.
So, an update to that to match what we have currently would be a simple solution, albeit temporary (potential issue whenever they go out of sync, so it'd become part of maintenance).

[Nuck-TH]

Updating file will still be mitigation of the symptom with possibility of data loss on any desync(like now).

I woud rather insist on checking if machinery in browser is working correctly, because for me it seems that it isn't - stuff breaks so hard that it can't both recover previous database or even create a new one(bak and bak-recover file stay, but no sqlite is created).

[Moonchild]

Look, that's not how this works. The extension has to be compatible with the browser, not the other way around. Expecting us to account for stuff extensions can possibly do in the core is not feasible, exactly because we allow extensions full access to browser internals. If an extension makes database entries that do not align with what the core structurally expects, things can (and as you've seen, will) break. That isn't a bug in the core, but rather an issue with how access to browser internals needs to be handled carefully.

The machinery in the browser is working correctly. It handles cookies exactly as it should. The only thing that could potentially be done is building some additional checks to make sure that cookies set from extensions are sane, but that would also not be a solution because then the extension would break as it can't set cookies it expects to be able to.
The solution has to be that the extension is adjusted to deal with this, and not rely on a potentially out-of-sync TLD list. Since this list isn't updated regularly the easy solution is to make sure it stays up-to-date whenever the list in the core is updated; which is what I meant with "maintenance". The reason it didn't break before is because our list in the platform had not been updated for a long time. If a different approach than reading the list from an extension resource can't be done easily, then a potential workaround would be to include the eTLD list inside the browser as an accessible resource:// url explicitly to cater to the extension wanting a copy of it, and use that so it's always in sync with browser internals, but otherwise nothing can reasonably be done here.

[Nuck-TH]

Updating file in extension did nothing. As expected actually - i think it is used for filter processing, not updates(because there is no real reason to need it there).

[Moonchild]

Updating file in extension did nothing. As expected actually -

Not really as-expected, at all. I would actually expect this obvious thing to be the cause. Hence my "bingo" when I saw it. It makes sense when it's updated in the platform causing a discrepancy for it to cause issues in an extension using its own copy of the list...

Well then, if it is, in fact, not the cause, then we need to figure out why this updating goes wrong, because I similarly don't see a need to touch cookies while updating a filter list. I would expect updating a filter list to at least involve some processing (which would involve parsing URLs and in turn determining domain portions), in turn calling upon the eTLD inside the extension. Why else would it be included in the extension to begin with? Maybe not the download itself but it'd be processed immediately after.

Alternatively, perhaps this URL the update is pulled from tries to set cookies and it goes wrong there? How exactly is this update performed in the extension? Are there any indicators inside the browser console when the update is attempted?

[satrow]

Out of default filters updating only one triggers the issue:
Malware domains > Online Malicious URL Blocklist

That's broken in JustOff's uBO version (dead link).

Can you test this: https://gitlab.com/malware-filter/urlha ... -blocklist ?

[Nuck-TH]

Can you test this: https://gitlab.com/malware-filter/urlha ... -blocklist ?

Test what exactly?
Point is not that link is dead and list doesn't update, but that attempt to do so triggers the discussed issue.

[ron_1]

Point is not that link is dead and list doesn't update, but that attempt to do so triggers the discussed issue.

Doesn't unchecking the box for that particular filter fix the issue?