importing bookmarks

Talk about code development, features, specific bugs, enhancements, patches, and similar things.
Forum rules
Please keep everything here strictly on-topic.
This board is meant for Pale Moon source code development related subjects only like code snippets, patches, specific bugs, git, the repositories, etc.

This is not for tech support! Please do not post tech support questions in the "Development" board!
Please make sure not to use this board for support questions. Please post issues with specific websites, extensions, etc. in the relevant boards for those topics.

Please keep things on-topic as this forum will be used for reference for Pale Moon development. Expect topics that aren't relevant as such to be moved or deleted.
person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

importing bookmarks

Unread post by person45 » 2019-08-21, 18:50

1. Pale Moon could not read a places.sqlite file, and then Pale Moon renamed it to places.sqlite.corrupt.

2. Chrome gave an option to import from Internet Explorer or Firefox. I moved places.sqlite to the Firefox profile. When I imported the places.sqlite file to Chrome, it loaded quickly. Chrome was then able to export an 11.2 mb html backup instantaneously.

3. This html backup could then be imported back to Pale Moon, but it took a long time with CPU usage. I even got an alert that a bookmark overlay script was busy or not responding.

4. Eventually the 11.2 mb html backup was turned into a 20 mb places.sqlite file by Pale Moon. This was a fix for the corrupted places.sqlite issue.

This whole process was unpleasant. Can you make Pale Moon just get it right the first time? I know bookmarks function is low on the priority list, but there is a clear difference in speed and capability. This is an area where improvement can be made. Is it easy to just copy whatever Chrome is doing here?

Also when I open the bookmarks menu for the first time during a session, I notice it takes a few seconds to open up on my physical hard drive. It feels like Pale Moon is retrieving the entire list before displaying. Would it be more efficient and faster if Pale Moon loaded only the first 30 entries quickly? The other entries can be retrieved as needed as I scroll down the list of bookmarks. (It's much faster on an ssd, so I understand if this is not considered to be an issue.)
Last edited by person45 on 2019-08-21, 19:46, edited 2 times in total.

User avatar
therube
Board Warrior
Board Warrior
Posts: 1650
Joined: 2018-06-08, 17:02

Re: importing bookmarks

Unread post by therube » 2019-08-21, 19:09

(While you may be able to throw around places.sqlite between different browsers & browser versions, including going from newer to older & or different, you can't really expect "compatibility".

If I had the need to move places.sqlite from 1 to another, I would certainly try it, but I would not be perturbed if it failed, or if it only worked to a point. Many times I may be glad that it worked - to a point.)


Where did the particular places.sqlite file originate from?
And what versions of FF (Quantum or Legacy?) / Chrome are we talking?


Different "import" methods will give differing import speeds.
Being able to read the .sqlite directly, is just that.
Importing certainly can be time consuming as that data has to be parsed & then written out to a .sqlite. And there can be time differences between .html & .json too.

And you happen to have a rather large (compared to myself) set of bookmarks (&/or other histories in your places.sqlite). (The size of a .sqlite is not particularly indicative of how much content it actually contains as they are generally created with a rather large initial size, then tend to grow by set increments.)

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-08-21, 19:38

therube wrote:
2019-08-21, 19:09
they are generally created with a rather large initial size, then tend to grow by set increments.)
I appreciate the fast response, but I already found this information about the places.sqlite size.

https://support.mozilla.org/en-US/questions/969977
"The minimum file size of the places.sqlite file is 10 MB and Firefox increases this file in 10 MB chunks when necessary (its size is never reduced when deleting history or bookmarks). This is done for performance issues caused by fragmentation."

I know it's preferable not to use the places.sqlite file directly, but I think I had a problem with my physical hard drive, and my older Pale Moon wasn't functioning properly.

This still doesn't excuse the fact that the backup html file that Chrome created took such a long time to transfer. Also I find it strange when Chrome is able to read the "corrupted" places.sqlite file while Pale Moon cannot.

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-08-21, 19:54

therube wrote:
2019-08-21, 19:09
If I had the need to move places.sqlite from 1 to another, I would certainly try it, but I would not be perturbed if it failed, or if it only worked to a point. Many times I may be glad that it worked - to a point.)
For reference, this is where I found the fix.

https://support.mozilla.org/en-US/questions/969977
"Then I tried to import all browsing history from Chrome, and guess what!

All browsing history is back in the non corrupted brand new data base.
Earlier I had mentioned that from chrome I was able to import browsing history from firefox , even though the data base of firefox was corrupt, chrome was successful in importing all of my firefox's history, so later in the new profile , all I had to do was reverse the process."

I'm not the only person reporting a compatibility issue with places.sqlite. I hope I don't come across as a whining person. I want Pale Moon to be the best it can be, and sometimes we can look to other browsers for ideas for improvement.

User avatar
Tomaso
Board Warrior
Board Warrior
Posts: 1622
Joined: 2015-07-23, 16:09
Location: Norway

Re: importing bookmarks

Unread post by Tomaso » 2019-08-21, 20:19

person45 wrote:
2019-08-21, 18:50
3. This html backup could then be imported back to Pale Moon, but it took a long time with CPU usage. I even got an alert that a bookmark overlay script was busy or not responding.
This is a very old and known Mozilla bug.
Here's my issue report @ GitHub:
https://github.com/MoonchildProductions/UXP/issues/505/
It was actually closed yesterday, for "being stale".

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-08-21, 20:23

Tomaso wrote:
2019-08-21, 20:19
This is a very old and known Mozilla bug.
I just tested restoring a json backup, and it still takes a long time with steady CPU usage and increased memory usage.

I know this is unimportant as long as it works eventually, but this is a small area where improvement would be very helpful.

Maybe this issue only affects users who have a big bookmarks "library" of links that were collected over the years.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35476
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: importing bookmarks

Unread post by Moonchild » 2019-08-21, 21:17

Tomaso wrote:
2019-08-21, 20:19
It was actually closed yesterday, for "being stale".
Obviously nobody had any interest in doing anything for this to improve it. Feel free to reopen it if you've done some work that should be evaluated.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-08-22, 17:55

Moonchild wrote:
2019-08-21, 21:17
Obviously nobody had any interest in doing anything for this to improve it. Feel free to reopen it if you've done some work that should be evaluated.
That's understandable. Browsing is much more important than bookmarking.

It's a little sad that Google can afford to just pay off a recent college graduate to take care of little features while the main work under the hood can be done by senior engineers.

https://www.theguardian.com/technology/ ... erage-2018
The average pay was over $200,000 in the UK.

It would be a good idea to try to increase market share if maybe on first use Pale Moon would automatically scan for other browsers and offer to import bookmarks with one click of a button. Making the migration process easy and fast from Firefox, Opera, Safari, Chrome, or Edge would be helpful. That's probably why Chrome has such good compatibility with even a corrupt places.sqlite.

http://kb.mozillazine.org/Locked_or_dam ... ces.sqlite
"It is also possible that Firefox was not closed properly, as might happen when your computer loses power with Firefox running, which can cause a corrupt places.sqlite file."

I'm not quite sure what happened, but maybe I had a computer crash or power outage which caused it.

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-08-22, 21:34

https://github.com/MoonchildProductions ... ssues/587/
Lorienna wrote, "importing large amounts of bookmarks from HTM files will result in PM hanging, and potentialy giving a "Unresponsive script" error message I have a lot of bookmarks too, and so far haven't seen the "Unresponsive script" error on PM, but it shows up on FF quite often, particularly on more recent releases. However, the HTML importing process takes rather long time indeed,"

tomaso wrote, "my tests now indicate that the issue first seems to have appeared between Firefox v23.0a1 (2013-05-13) and v23.0 Final."

It seems there's some kind of issue introduced in the year 2013.

User avatar
Tomaso
Board Warrior
Board Warrior
Posts: 1622
Joined: 2015-07-23, 16:09
Location: Norway

Re: importing bookmarks

Unread post by Tomaso » 2019-08-23, 10:34

person45 wrote:
2019-08-22, 21:34
Lorienna wrote, "importing large amounts of bookmarks from HTM files will result in PM hanging, and potentialy giving a "Unresponsive script" error message
Actually, I wrote that.
Lorienna simply quoted it. :)

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-09-06, 02:53

I know a way to test this.

Generate a list of URLs that don't exist:
http://bookmarktest.com/1
http://bookmarktest.com/2
http://bookmarktest.com/3

Until you get a 5mb html file. Then go to Show All Bookmarks>Import and Backup>Import Bookmarks from HTML...

Compare the time it takes to create the places.sqlite file with how fast Chrome or Opera does it.

https://support.mozilla.org/en-US/questions/1100618
"Firefox seems to using JavaScript code to import an HTML file and that is always slower than using C++ code and if this JavaScript code runs into the check for hangs (max_chrome_script_run_time) then you get a slow script alert. Importing a large amount of bookmarks can take considerable time."

"I did notice a considerable difference between importing in Opera and Firefox when all went well : roughly 1 second and 1 minute for the same file."

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35476
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: importing bookmarks

Unread post by Moonchild » 2019-09-06, 07:21

Maybe a separate tool should be created to import large amounts of bookmarks into places.sqlite that provides a more interactive experience.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Goodydino
Keeps coming back
Keeps coming back
Posts: 820
Joined: 2017-10-10, 21:20

Re: importing bookmarks

Unread post by Goodydino » 2019-09-15, 20:37

person45 wrote:
2019-08-21, 20:23
I just tested restoring a json backup, and it still takes a long time with steady CPU usage and increased memory usage.

I know this is unimportant as long as it works eventually, but this is a small area where improvement would be very helpful.

Maybe this issue only affects users who have a big bookmarks "library" of links that were collected over the years.
How many thousand bookmarks do you have? I just restored my bookmarks after backing them up and it took less than a second. I backup bookmarks to a folder outside of the profile as .json.

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-10-01, 19:52

Goodydino wrote:
2019-09-15, 20:37
How many thousand bookmarks do you have?
I don't know, but if I bookmark 5 per day over 6 years, then that's about 11,000 without much effort. It just steadily grows over time.

I realize this affects only a small percentage of users, so I can accept that it is a low-priority issue.

It's not the Pale Moon developers' fault. The "cor-el" guy at the firefox forum said that it was a javascript issue. Mozilla may have messed it up in the year 2013. Tomaso wrote, "my tests now indicate that the issue first seems to have appeared between Firefox v23.0a1 (2013-05-13) and v23.0 Final." If it's possible to revert back to the old way, then maybe that would be a much easier fix.

User avatar
moonbat
Knows the dark side
Knows the dark side
Posts: 4942
Joined: 2015-12-09, 15:45
Contact:

Re: importing bookmarks

Unread post by moonbat » 2019-10-02, 01:11

person45 wrote:
2019-10-01, 19:52
Goodydino wrote:
2019-09-15, 20:37
How many thousand bookmarks do you have?
I don't know, but if I bookmark 5 per day over 6 years, then that's about 11,000 without much effort. It just steadily grows over time.
More importantly, how often do you revisit the sites you've bookmarked? The visit date/visit count fields in the bookmark manager should be instructive. For comparison, my bookmarks backup from del.icio.us over a 12 year period is just under 1500.
"One hosts to look them up, one DNS to find them and in the darkness BIND them."

Image
Linux Mint 21 Xfce x64 on HP i5-5200 laptop, 12 GB RAM.
AutoPageColor|PermissionsPlus|PMPlayer|Pure URL|RecordRewind|TextFX

coffeebreak
Moon Magic practitioner
Moon Magic practitioner
Posts: 2986
Joined: 2015-09-26, 04:51
Location: U.S.

Re: importing bookmarks

Unread post by coffeebreak » 2019-10-02, 02:17

person45 wrote:
2019-10-01, 19:52
I don't know, but if I bookmark 5 per day over 6 years, then that's about 11,000

person45
Fanatic
Fanatic
Posts: 104
Joined: 2017-10-20, 07:00

Re: importing bookmarks

Unread post by person45 » 2019-10-02, 02:18

moonbat wrote:
2019-10-02, 01:11
More importantly, how often do you revisit the sites you've bookmarked?
That's a good point. Most of my bookmarks are a collection of sites of interest, research, or information. I guess I use it more like an archive.

This is another reason why it's a low-priority issue.

But the question is why would Mozilla use this slower, javascript method in the 1st place? Is there any benefit to it?

Goodydino
Keeps coming back
Keeps coming back
Posts: 820
Joined: 2017-10-10, 21:20

Re: importing bookmarks

Unread post by Goodydino » 2019-10-02, 21:09

If you never check the validity of the bookmarks, you are likely to have at least a few dozen invalid ones. Web pages are not eternal.

Locked