Can't backup bookmarks or run integrity check

Talk about code development, features, specific bugs, enhancements, patches, and similar things.
Forum rules
Please keep everything here strictly on-topic.
This board is meant for Pale Moon source code development related subjects only like code snippets, patches, specific bugs, git, the repositories, etc.

This is not for tech support! Please do not post tech support questions in the "Development" board!
Please make sure not to use this board for support questions. Please post issues with specific websites, extensions, etc. in the relevant boards for those topics.

Please keep things on-topic as this forum will be used for reference for Pale Moon development. Expect topics that aren't relevant as such to be moved or deleted.
Dariush

Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-13, 17:14

I have a largish (57k entries) slightly-corrupted places.sqlite from a two-year-old profile. By "slightly corrupted" I mean that there is some weird behavior like search returning a limited number of entries, places.sqlite being renamed as places.sqlite-corrupt from time to time, general sluggishness and, notably, an inability to export bookmarks as HTML or backup as JSON - simply nothing happens when I choose the path to save to. When I go to about:support and press "verify integrity", Pale Moon finds some problems (the same thing happens when I run pragma integrity_check on sqlite DB itself), but I can access the bookmarks themselves more or less fine. I fixed the corruption by running ".clone"; the new version passes sqlite's integrity test and doesn't have any other previous problems, except for two - I can't backup to JSON (but I can export to HTML) and when I click "verify integrity" in about:support, nothing happens.

I realize that the information provided above may be limited, but I don't know what other info I can provide short of uploading the whole DB.

This may sound minor (and, well, it is), but what this means is that without JSON backup I can't transfer the bookmarks _with their annotations_ (which is the important part) to my current profile, so finding a different way to do this can also work for me.

vannilla
Moon Magic practitioner
Moon Magic practitioner
Posts: 2183
Joined: 2018-05-05, 13:29

Re: Can't backup bookmarks or run integrity check

Unread post by vannilla » 2018-12-13, 18:01

This advice isn't really going to be helpful, but if everything else fails...
Start two Pale Moon instances with two different profiles (if I understand correctly, the one with the corrupter places.sqlite and the current one are different), and manually create each bookmark in the destination profile, copying the data from the source profile.
Since you say the browser can access the bookmarks, at least we know this method will work.
Last edited by vannilla on 2018-12-13, 18:01, edited 1 time in total.

Dariush

Re: Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-13, 18:16

Not only is that completely unfeasible for 57k bookmarks for obvious reasons, but also that doesn't even copy over the annotations, which is the whole point.

vannilla
Moon Magic practitioner
Moon Magic practitioner
Posts: 2183
Joined: 2018-05-05, 13:29

Re: Can't backup bookmarks or run integrity check

Unread post by vannilla » 2018-12-13, 20:37

Well, if you do a small batch (say, 20 bookmarks) every day, in a year maybe you're done.
And if annotations are not copied with the "copy" command, manually annotate the new bookmark.
Yeah, it's a pretty useless advice, but if no other solution is found, you have nothing else but this.

JustOff

Re: Can't backup bookmarks or run integrity check

Unread post by JustOff » 2018-12-13, 21:16

You can try the following (backup your profile first):

1) Install the Speed Start add-on
2) Open and then close a new tab - this will create a service folder named SStart in the Bookmarks Menu
3) Open the Library (Ctrl+Shift+B), create a folder inside the SStart and name it EXPORT
4) Move all your bookmarks into this EXPORT folder
5) Open the Add-ons Manager (Ctrl+Shift+A), then in the Speed Start options select the Extra tab and click Save to create a backup

If everything goes well you will be able to restore the Speed Start backup into the new profile and move your bookmarks from the SStart/EXPORT back to their place. If you fail to save all bookmarks at once, you can try to do it in several stages, moving them to the SStart/EXPORT folder in parts.

Please note that currently the Speed Start backup does not preserve the dates of creation and last visit, the visit count, tags and keywords, but only the name, address and annotation.

Dariush

Re: Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-14, 01:22

Thanks, but didn't work - annotations are gone.

To clarify, by annotations I mean the contents of "moz_items_annos" table, where an addon called TooManyTabs stores the history of bookmarks, which is a must-have for me. They are exported when doing "Backup" from Library (which creates a JSON, and which doesn't work on the decorrupted places.sqlite), but not when doing "Export bookmarks to HTML" (which works).
Last edited by Dariush on 2018-12-14, 01:28, edited 2 times in total.

RJARRRPCGP
Lunatic
Lunatic
Posts: 400
Joined: 2015-06-22, 19:48
Location: USA (North Springfield, Vermont)
Contact:

Re: Can't backup bookmarks or run integrity check

Unread post by RJARRRPCGP » 2018-12-14, 01:50

Did you have any disorderly Windows shutdowns or reboots recently? (An improper shutdown, shutdown without using the shutdown command or a bugcheck-related reboot.)

Dariush

Re: Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-14, 02:14

Er, no. What does that have to do with this issue?
Last edited by Dariush on 2018-12-14, 02:14, edited 2 times in total.

RJARRRPCGP
Lunatic
Lunatic
Posts: 400
Joined: 2015-06-22, 19:48
Location: USA (North Springfield, Vermont)
Contact:

Re: Can't backup bookmarks or run integrity check

Unread post by RJARRRPCGP » 2018-12-14, 02:53

Dariush wrote:Er, no. What does that have to do with this issue?
I had profile corruption and the only other thing that jumps out on my end, were unexpected shutdowns, because of power failures and of course, Windows 7 gave me a "critical" category event log error, because of that.
I was forced to wipe out my profile, because it caused a web site to look like there was an engine problem, or that PM's built-in CSP blocked essential functionality to candelacorp.com

I was of course suspecting an incomplete write to the database file, because of power loss or a crash. I apologize.

Galaxy
Moonbather
Moonbather
Posts: 51
Joined: 2016-07-04, 03:41

Re: Can't backup bookmarks or run integrity check

Unread post by Galaxy » 2018-12-14, 22:17

Have you used the export function in TooManyTabs (Options > Advanced) to export some of your bookmarks with annotations to a file? Then use the import function in TooManyTabs to import those bookmarks into your current profile. See if this works and if the annotations are still there.

Dariush

Re: Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-15, 18:33

Galaxy wrote:Have you used the export function in TooManyTabs (Options > Advanced) to export some of your bookmarks with annotations to a file? Then use the import function in TooManyTabs to import those bookmarks into your current profile. See if this works and if the annotations are still there.
Thanks for the idea, but it didn't work. TMT didn't export the annotations.

User avatar
Isengrim
Board Warrior
Board Warrior
Posts: 1325
Joined: 2015-09-08, 22:54
Location: 127.0.0.1
Contact:

Re: Can't backup bookmarks or run integrity check

Unread post by Isengrim » 2018-12-15, 18:53

Does FEBE export annotations?
a.k.a. Ascrod
Linux Mint 19.3 Cinnamon (64-bit), Debian Bullseye (64-bit), Windows 7 (64-bit)
"As long as there is someone who will appreciate the work involved in the creation, the effort is time well spent." ~ Tetsuzou Kamadani, Cave Story

Dariush

Re: Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-19, 02:26

Isengrim wrote:Does FEBE export annotations?
FEBE doesn't work period, tested on a clean profile (well, not counting TMT). Selected "backup on startup", set directory and what to backup, restarted. Backup "ran", but no files appeared. Selected "backup on shutdown" with prompt, tried to close PM, prompt appeared, neither OK or Cancel buttons did anything except refresh the prompt, I had to shut down the task process. Thanks, but no, not touching this.
Last edited by Dariush on 2018-12-19, 02:27, edited 2 times in total.

Dariush

Re: Can't backup bookmarks or run integrity check

Unread post by Dariush » 2018-12-24, 22:59

Well, I managed to manually merge the two places.sqlite. No problems so far. In case anyone stumbles upon this post in the future:

Code: Select all

-- Ensure that new ids don't conflict with old ones
update moz_places set id = id + 1000000, favicon_id = favicon_id  + 1000000;
update moz_favicons set id = id + 1000000;
update moz_items_annos set id = id + 1000000, item_id = item_id + 1000000;
update moz_bookmarks set id = id + 1000000, fk = fk + 1000000, parent = parent + 1000000;

-- Root folders will be placed directly in Bookmarks Menu
update moz_bookmarks set parent = 2 where parent = 1000001;

-- Tweak path to the other places.sqlite
ATTACH "places-2.sqlite" AS target;

-- We are only interested in annotations that have the name "TooManyTabs.visibotech.com/tmt/tabState"; delete the other ones and tweak attribute ids
-- (they are likely going to be different for you)
delete from moz_items_annos where anno_attribute_id != 18;
update moz_items_annos set anno_attribute_id = 2;

-- Update duplicate favicon entries so bookmarks point to the ids in the other DB
CREATE TEMPORARY TABLE temp AS select f.id as old_id, f2.id as new_id from moz_favicons as f join target.moz_favicons as f2 on f.url = f2.url;
update moz_places set favicon_id = (select new_id from temp where moz_places.favicon_id = old_id) where favicon_id in (select old_id from temp);

-- Finally merge the useful tables
INSERT INTO target.moz_places SELECT * FROM moz_places;
INSERT INTO target.moz_bookmarks SELECT * FROM moz_bookmarks;
INSERT INTO target.moz_items_annos SELECT * FROM moz_items_annos;
INSERT or ignore INTO target.moz_favicons SELECT * FROM moz_favicons;

Locked