Bookmarks db slow with many items

[KSAKS]

Win7 x64 - fully updated, but it is not OS related problem:
34.0.0 - but this is problem for previous versions as well:
64-bit:

Well as my bookmarks db is growing I'm noticing more and more the browser is not well optimized for very large bookmark databases - I have around 18500 bookmarks atm. Recently for example I needed to sort one bookmarks subfolder with around 2800 items and it took over 15-20 minutes and the RAM usage during the operation was over 17GB just for the browser. The browser has been unresponsible during the whole operation, but the CPU usage has been only around 7%. Also I've noticed, if I have the browser running for longer time, I cannot anymore move the bookmarks between bookmarks subfolders and I need to restart the browser first. I know this is probably no so common to have so many bookmarks, but I need it this way. It is something I can live with, but would be really nice if this is possible to optimize somehow.

[back2themoon]

Hello,

This may be related to my age-old issue with bookmarks losing their custom sorting during an export process. Since you have many bookmarks, please perform this test if possible:

1. Back up your current bookmarks!
Safest methods, either: a) Back up your entire profile, or b) Close PM and copy these two files in a safe folder: places.sqlite and sessionstore.js (you can simply close PM and put them back later to restore).

2. Library/Import and Backup/Backup... and save the resulting .json file. I believe this action falls apart with many bookmarks and/or complex structures.

3. Library/Import and Backup/Restore/Choose File... and restore the exported .json file.

4. Verify if bookmarks structure and sorting is retained e.g. folder positions and bookmark items inside folders.

Thanks

[KSAKS]

Hello,

This may be related to my age-old issue with bookmarks losing their custom sorting during an export process. Since you have many bookmarks, please perform this test if possible:

1. Back up your current bookmarks!
Safest methods, either: a) Back up your entire profile, or b) Close PM and copy these two files in a safe folder: places.sqlite and sessionstore.js (you can simply close PM and put them back later to restore).

2. Library/Import and Backup/Backup... and save the resulting .json file. I believe this action falls apart with many bookmarks and/or complex structures.

3. Library/Import and Backup/Restore/Choose File... and restore the exported .json file.

4. Verify if bookmarks structure and sorting is retained e.g. folder positions and bookmark items inside folders.

Thanks

Hello, I can reply to this test even without doing it again. This is process I did many times in the past and last time I did it was yesterday when I was cleaning my Palemoon profile and setting up everything from scratch after maybe 10-15 years. Maybe I should also mention I'm using portable Palemoon version. I didn't notice problems with exporting so many bookmarks to the json file and importing it back. The bookmarks structure is kept for me. Might be something specific to your sorting or some characters. Only thing not kept is the websites icons. That is restored only after I visit the bookmark again in the browser. The resulting json file after the export is 10.2 MB big and that is imho the main problem. It is probably something which comes from Mozilla code, because I remember when I needed to work with very big json files inside TOR Browser, there was as well huge memory usage and slowdown of everything.

[back2themoon]

Thanks for the report. I can't pinpoint this sorting issue.

With ~2800 bookmarks here, performance is ok.

[Goodydino]

Win7 x64 - fully updated, but it is not OS related problem:
34.0.0 - but this is problem for previous versions as well:
64-bit:

Well as my bookmarks db is growing I'm noticing more and more the browser is not well optimized for very large bookmark databases - I have around 18500 bookmarks atm. Recently for example I needed to sort one bookmarks subfolder with around 2800 items and it took over 15-20 minutes and the RAM usage during the operation was over 17GB just for the browser. The browser has been unresponsible during the whole operation, but the CPU usage has been only around 7%. Also I've noticed, if I have the browser running for longer time, I cannot anymore move the bookmarks between bookmarks subfolders and I need to restart the browser first. I know this is probably no so common to have so many bookmarks, but I need it this way. It is something I can live with, but would be really nice if this is possible to optimize somehow.

Are you sure all those bookmarks are valid? Do you check them with an extension that can do that? I find that when I check my bookmarks every few weeks or so, that some sites have disappeared. I delete those, since dead bookmarks are useless. I have hundreds of bookmarks, not thousands, probably for that reason.

[KSAKS]

Are you sure all those bookmarks are valid? Do you check them with an extension that can do that? I find that when I check my bookmarks every few weeks or so, that some sites have disappeared. I delete those, since dead bookmarks are useless. I have hundreds of bookmarks, not thousands, probably for that reason.

No, I'm not checking them with such extension and I'm 100% sure some of the bookmarks are dead of course, but such bookmarks are even more valuable for me. Because that is stuff I cannot find anymore with regular search engines, but in most of the cases I can still access them with web.archive.org or archive.ph if I know the original url. There are tons of interesting and important articles I collected during last ~20 years I can still access this way in few moments directly from my browser.

[Gemmaugr]

I have 4000 bookmarks and experience no such issue, though it's probably because they're all in folders and sub-folders (and sub-sub-folders etc).
When you say that you "needed to sort" them, do you use the menu bar dropdown or go into Organize Bookmarks (ctrl+shift+b), or something else/addon?

I completely agree that dead bookmarks are even more important than live ones, since you can indeed use them to find the content elsewhere instead.

[KSAKS]

I have 4000 bookmarks and experience no such issue, though it's probably because they're all in folders and sub-folders (and sub-sub-folders etc).
When you say that you "needed to sort" them, do you use the menu bar dropdown or go into Organize Bookmarks (ctrl+shift+b), or something else/addon?

I completely agree that dead bookmarks are even more important than live ones, since you can indeed use them to find the content elsewhere instead.

Both menu bar dropdown and go into Organize Bookmarks (ctrl+shift+b). I'm not using any addon for the sort.

[KSAKS]

Ok guys looks like I have found the core of this problem. I did mistake in my testing environment (I have here multiple PM portables for tests while I'm setting up everything from scratch) and forgot there active one extension used for another tests and that is the main cause of those slowdowns. Without it is just "normally" little bit slower (maybe 30-40 seconds freeze while importing the 10.2MB json file and few seconds freeze on actions, so still some optimizations would be nice) but not extreme slowdown like I'm describing above.

So the core of the extreme slowdown is this extension https://addons.palemoon.org/addon/update-alert/. Looks like it is not properly handling only the items inside "Update Scanner's Pages", but somehow processing all bookmarks if they are resorted.

[Gemmaugr]

Ok guys looks like I have found the core of this problem. I did mistake in my testing environment (I have here multiple PM portables for tests while I'm setting up everything from scratch) and forgot there active one extension used for another tests and that is the main cause of those slowdowns. Without it is just "normally" little bit slower (maybe 30-40 seconds freeze while importing the 10.2MB json file and few seconds freeze on actions, so still some optimizations would be nice) but not extreme slowdown like I'm describing above.

So the core of the extreme slowdown is this extension https://addons.palemoon.org/addon/update-alert/. Looks like it is not properly handling only the items inside "Update Scanner's Pages", but somehow processing all bookmarks if they are resorted.

Thanks for reporting back. I use..well, have, that addons myself. Though I only really use it for one site. Time to remove it. Thanks again!

[KSAKS]

Thanks for reporting back. I use..well, have, that addons myself. Though I only really use it for one site. Time to remove it. Thanks again!

No problem. I have 9 items inside it atm. So I guess it is not about how many items you have there, but about the amount of total bookmarks in the browser and if it is not extreme like in my case, you will not notice it. Unfortunately I cannot remove this addon. It is essential for me to check updates on few pages and I don't know about alternative. I've tried also with original Update Scanner addon and it is the same (maybe even worse). So that seems to be really old bug.

[back2themoon]

Are you sure all those bookmarks are valid? Do you check them with an extension that can do that?

Any suggestions for such extensions?

[Mæstro]

I completely agree that dead bookmarks are even more important than live ones, since you can indeed use them to find the content elsewhere instead.

When this happens, I update the rotten link instead to point to an archived version. About two out of every five bookmarks I keep is like this.

[Goodydino]

Are you sure all those bookmarks are valid? Do you check them with an extension that can do that?

Any suggestions for such extensions?

I like CheckPlaces. You can get it from this site: http://daniel-lange.com/archives/71-Syn ... d-ons.html. There is also Bookmarks Checker which you can get from the Classic Add-ons Archive. That one does not have quite as many options however.

[Gemmaugr]

I completely agree that dead bookmarks are even more important than live ones, since you can indeed use them to find the content elsewhere instead.

When this happens, I update the rotten link instead to point to an archived version. About two out of every five bookmarks I keep is like this.

That isn't always possible sadly. Wayback Machine has started to purge some sites, and Archive.ph requires google captcha if it even has the page in question.
I usually use the title/topic of the dead bookmark to find another instance that talks or shows the content in question instead.

[Mæstro]

Wayback Machine has started to purge some sites, and Archive.ph requires google captcha if it even has the page in question.

Archive.ph’s problems are familiar to me. Its Cloudflaring has led me to avoid it. But what has been happening to the Archive? I have never personally observed sites being removed from it, as opposed to having been deliberately excluded from the start. More information here would help.

[Gemmaugr]

Wayback Machine has started to purge some sites, and Archive.ph requires google captcha if it even has the page in question.

Archive.ph’s problems are familiar to me. Its Cloudflaring has led me to avoid it. But what has been happening to the Archive? I have never personally observed sites being removed from it, as opposed to having been deliberately excluded from the start. More information here would help.

https://web.archive.org/web/20260000000 ... 25-edition

https://archive.org/post/778/exclusions ... ck-machine

https://wiki.archiveteam.org/index.php/ ... ck_Machine

[Mæstro]

Ah, these are just the Archive’s policies which I have always known about, and which have remained unchanged. I misinterpreted ‘purge’ to suggest the Archive has lately deleted many formerly safe sites in a concentrated push, which would be quite alarming. The situation is not so scary as it seemed.

[back2themoon]

Thanks for the info, Goodydino.

As for dead bookmarks, I'd definitely prefer downloading/saving those pages offline, rather than relying on Wayback Machine.

[KSAKS]

That isn't always possible sadly. Wayback Machine has started to purge some sites, and Archive.ph requires google captcha if it even has the page in question.
I usually use the title/topic of the dead bookmark to find another instance that talks or shows the content in question instead.

I didn't notice purging of sites from archive.org and also I don't have problem with google captchas on archive.ph, but this might be different for various websites and various user locations.

Thanks for the info, Goodydino.

As for dead bookmarks, I'd definitely prefer downloading/saving those pages offline, rather than relying on Wayback Machine.

Yes, I'm doing sometimes as well local copies, but the additional value of archive.ph or archive.org copies is you can send such link to anybody and it is proof such url and content really existed in the past. Locally archived copy is not a proof, because anybody can fake it easily.