Obsolete unlisted message after deleting attachment

Former board for discussion of BinOC Applications and Extensions using the Unified XUL Platform.
For historical reference.

Moderator: athenian200

Forum rules
If you are interested in forking any BinOC project and want to coordinate such an effort through the forum, please use viewforum.php?f=69
User avatar
Bilbo47
Fanatic
Fanatic
Posts: 234
Joined: 2017-11-18, 04:24

Obsolete unlisted message after deleting attachment

Unread post by Bilbo47 » 2020-12-29, 04:38

TL;DRs:
1 - Deleting attachments can result in unlisted messages.
2 - Under [Server Settings] / [When I delete a message], how are the three options supposed to work and interact with Expunge?
Detail:
When you delete an attachment from a message, the message list in IL is updated to show the message's new size. On the IMAP server side, the folder for that mailbox now contains an extra file, which apparently is the original message file before starting the attachment deletion. The old message-file is properly not shown in IL, while the new smaller message-file *is* shown.

This mismatch between number of files and number of messages is temporary. After you exit IL and wait for its TCP connections to close and for its process to unload from memory, the IMAP folder now again contains only one file for each message in the mailbox, so the original message-file has been deleted.

A problem arises when IL does not exit normally for whatever reason, before the user gets a chance to exit normally: the message-file to be deleted is not deleted, and just gets left there.

Let me be clear: IL does *not* crash on me. Rather, my internet/WiFi goes down/gets interrupted, and the VPN is set to abort a list of processes; these are all the programs that make external IP connections. The VPN program makes very sure that no non-VPN connections are established, by simply ending those programs that could do so when its connection drops. From the point of view of IL's data, this is the same as if IL had crashed: after the VPN comes back online and IL gets restarted, IL has to perform any necessary crash-recovery procedures. This happens several times a week, regardless of the weather outside. IL stays running all day, so it often stands a good chance of getting shut down hard, through no fault of its own.

Similar to a operations that update a message and result in a "previous message-file" that is marked for deletion ... when you delete a message in IL, it is moved to the Trash folder (if the Server Setting says so). From there it can easily be un-deleted. You can also "Empty Trash" on the folder. Or you can wait until IL exits, when the account's "Empty Trash on Exit" switch comes into play. As you might expect in a crash, "deleted" messages in the Trash folder will still exist the next time IL is started, because the "Empty Trash on Exit" step was never triggered. When IL does eventually exit normally, the Trash folder gets emptied out at that time (if the Server Setting says so).

Anyway, the same "delayed delete" thing happens with "old/original" message-files containing all original attachments: the file-to-be-deleted is not cleared out, because IL's shutdown sequence never fires. Except these message-files are not *in* a Trash folder, so they never get automatically (or manually) cleared out in the future. They are orphaned files, which take up space on the server but do not appear in IL's message list for that mailbox. They end up counting against the storage quota for the mail account, but they cannot be seen, let alone deleted, from within IL (if the Server Setting says so).

I dunno if this is a hole in the IMAP spec, or an oversight in TB/IL's implementation, or a bug in the IMAP server, or what.
I discovered this when my account reached 75% full, and it was time to delete large messages, or at least delete large attachments.

IMAPSize is great at doing low-level maintenance on your mail account's folders and files. I think it even supports mass-migration of messages between different accounts on different email providers/servers. For our purposes, it shows the total storage space taken up for each IMAP folder. You can drill down to sub-folders. "Get Message List" shows the list of messages inside a folder, with header details like dates and sizes. The details are displayed in a strikeout/strikethrough font when that message has an attribute of "to be deleted".

As another tool in my box, an FTP client shows the number of files in the mailbox folders (new / tmp / cur).

Comparing the three message lists, sorted by size descending, to find the largest messages in the folder. These are the message that need to be removed / attachments removed from the server, to bring down the storage space used. Except it was clear that IL was *not* showing the largest message in the folder! What it *did* show, was that same message, except with some attachments deleted. This would have been the result of my having deleted the attachment in the past, except the server side was for whatever reason not cleaned up properly, and IL was not listing the message-file. The other tools showed the server's view, and revealed the discrepancy.

One way of clearing the orphan message-files off the server is to use IL and move the listed messages to another folder, delete the folder that looks empty (but it's not), then close IL and let the IMAP cleanup happen; next re-open IL and rename/move the folders as needed to restore things to normal. The other tools then verify that the total storage used is smaller.

OR you could use IMAPSize to display the message-files marked for deletion, and either move or delete them via its IMAP interface.
The barbarian way would be to slash-and-burn - I mean - delete the files behind IMAP's back by using your FTP client.

The question then becomes: is there an IMAP-compliant way for IL to prevent these orphan files in the first place? Like, maybe the "old/original" message-file should be moved to the Trash folder, so that even if the cleanup phase is missed this time around, IL will sooner or later execute a normal shutdown, in which case the Trash would be emptied out, and any orphans from deleting attachments would be cleared, just like any other messages deleted by user.

Testing shows that changing the account's Server Setting for "When I delete a message" from "Move it to this folder" to "Remove it immediately" makes no difference; deleting an attachment still leaves the marked-for-deletion message-file in the folder. In contrast, a whole message deleted while this setting is in effect results in the message-file being marked for deletion, and the message is *not* moved to the Trash folder. I assumed that just like messages from which attachments have been deleted, message-files with this attribute would be deleted when IL exits normally; however testing showed they remained as orphans: taking up storage space but not appearing in IL's listing.

(About "Move it": when a folder is not specified, and the setting is left on [Choose Folder...], one assumes that IL will go ahead and default to moving deleted messages to the Trash folder.)

Not sure what to think about all this. That non-deletion is the behavior I would have expected from changing the setting to "Just mark it as deleted". In case this feature doesn't work right, small orphan messages wouldn't really matter, but inaccessible leftovers from trying to clear out large useless datas would surely be a problem. The unexpected behavior happened even when IL was started in Safe Mode with Add-Ons disabled.

Humph. Changing the Server Setting of "When I delete a message" from "Remove it immediately" to "Just mark it as deleted" and restarting IL resulted in the "orphan" messages showing up in IL's message list, except now with a strikethrough font and a red-X icon. The context menu had "Undelete message" in place of the usual "Delete message". The FTP view showed the "Undelete" operation just removes the "to-be-deleted" attribute from the message-file. The Delete key simply toggles the attribute on an off, so the strikethrough and icon appear and disappear. Then when IL exits, the marked files are still left there. How are those messages ever supposed to actually get deleted? Anyway, NOW the message-files containing the original pre-deletion attachments *do* show up in the message list. Again the Delete key simply un-marks the message for deletion, so now I have two copies of the message: one as I intend, and one with the deleted attachment still attached. Huh? These "to-be-deleted" messages still persist in IL even after moving them to the Trash and closing + re-opening IL. I don't get it.

This is a frustrating case of, "Here is one glitchy thing that I can deal with and work around in a satisfactory manner, now that I know about it" ... except when testing every feature that could possibly be related, it turns out that none of them work in the expected way. How can I rely on a workaround if the straightforward and supported methods are not even reliable? Are there bugs? Or documentation defects? Am I not understanding what the intended functionality is supposed to be? I *don't* want to go and re-test this stuff under TB. Sigh. It might come to that. I understand (but not expect) a limitation like "Changes to Server Settings don't take effect until IL is restarted" (this is frequently not the case), but I don't understand the rest of my discoveries here.

Update: testing on the top-level Inbox worked in a more reasonable way. Previous testing was on a folder three levels deep, with a space in the name. Ya never know how or whether these things affect the operation of features. Like, a major reason for moving from TB to IL was, my demanding usage was going beyond the limits of TB's capabilities. IL is much better about handling my requirements, but I won't be surprised if we find longstanding bugs inherited from old TB code, which had remained hidden/untested behind other more restrive limits, which have since been lifted/solved in IL, thus exposing the bugs to executing/manifesting for the first time.

As an aside, I remember reading something a long time ago that explained the difference between "Just mark it as deleted" versus "Remove it immediately". The article said they both work the same as far as setting the "to-be-deleted" attribute on the message-file (via IMAP) ... except "Remove it" simply suppressed displaying the marked message, while "Just mark it" continued to display the message, albeit in a format that shows the marking. The salient point was, the "Remove it" setting actually did nothing of the sort; mail clients had developed to present a user interface that *only represented* "I have deleted the message" while also following the IMAP spec - which basically disallowed/didn't support the client's actually deleting message-files from the server.

I dunno if the spec or implementation practice has changed since then; all I know is, the only way I know to *really* drop a message from the server using only IL is with "Move it to (Trash) folder" and then exit IL (or "Empty Trash"). Arg! Even the Trash folder's "Empty Trash" menu item does not appear when "Just mark it" is set. I see that "Expunge" actually removes the messages that are marked for deletion. Presumably under "Message Storage", the option "Clean up ('Expunge') Inbox on exit" would do as it says, and remove the marked messages in the Inbox folder when IL closes. What about other folders containing marked messages; does this switch cover folders other than Inbox? Is there a menu item to Expunge a folder manually? This variety of options has confused me for years.

This reference says "Just mark it as deleted: this setting keeps the deleted message in the folder, but sets a strikethrough line to show it has been marked as deleted. To clear the folder of deleted messages, right-click on the folder and select Compact." This might seem to indicate that the "Compact" menu item performs an IMAP "Expunge" operation on the folder. Except we know that compacting applies when "Message Store Type" is set to the MBox format, where all messages in a folder are stored in one file, and Compact means copy all non-deleted messages from that file to a new file. OTOH MailDir format stores each message in its own file. Thus, compacting does not apply to MailDir-style folders; this explains why "Compact" never appears as an option for my folders.

Is there an up-to-date Mozilla article or StackOverflow entry that explains how this part of IMAP is supposed to work conceptually? Hoping to avoid a slog through RFC-speak or a code review. For now I have to remember to close IL after folder maintenance, just to prevent non-deletion/orphanage when my net connection unexpectedly hiccups. Sorry for the rambling post; obviously today's coffee was too strong.

New Tobin Paradigm

Re: Obsolete unlisted message after deleting attachment

Unread post by New Tobin Paradigm » 2020-12-29, 06:03

Ok.. not even gonna read that due to length.

Look up CharmCityCrab and do the opposite of that. In other words tl;dr it for me.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35476
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Obsolete unlisted message after deleting attachment

Unread post by Moonchild » 2020-12-29, 09:14

Not going to read the entire essay but this stood out to me:
Bilbo47 wrote:
2020-12-29, 04:38
Rather, my internet/WiFi goes down/gets interrupted, and the VPN is set to abort a list of processes; these are all the programs that make external IP connections. The VPN program makes very sure that no non-VPN connections are established, by simply ending those programs that could do so when its connection drops. From the point of view of IL's data, this is the same as if IL had crashed:
That is a real poor man's job of handling this. Any system VPN has full control over network traffic and can simply suspend the network flows if the VPN connection drops (as any proper client should), not just kill processes. Hard-killing processes should never be done by choice. Ever. At the very least issue a proper shutdown command when doing so. But even then, as said this isn't necessary if network traffic is simply suspended as it should be doing in the first place. So... Don't blame Interlink for desynced data when you choose to kill it with a dirty shutdown.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Bilbo47
Fanatic
Fanatic
Posts: 234
Joined: 2017-11-18, 04:24

Re: Obsolete unlisted message after deleting attachment

Unread post by Bilbo47 » 2020-12-29, 23:20

[VPN kills IL sometimes]
That is a real poor man's job of handling this ... Don't blame Interlink for desynced data when you choose to kill it with a dirty shutdown.
Totally agree, thank you. Still leaves me looking for how to help IL recover "lost" messages when internet goes out, regardless of anything else.

Thinking out loud here, on how to find and delete "orphan" messages in IL. Change Server Settings so that messages marked for deletion are displayed in the message list. Manually tour all folders, checking for any messages that are displayed in strikethrough font. At this point, how to get IMAP to actually delete those message?

Okay, testing showed the situation does get worked out eventually. Recipe:
Using an account set to "Move deleted messages to this (default) folder", in a folder one level under Inbox.
Delete an attachment from a message. Using IMAPChk, confirm the 'hidden' message exists as marked for deletion. Kill IL's process.
Re-start IL. Confirm the hidden message-file still exists.
Using IL, select a different folder, re-select the folder under test, close IL normally.
In IMAPChk, the message that had been marked for deletion should no longer exist.

What this means is, the IMAP spec together with IL's implementation, executes the normal protocols of 'Get Message List' and later clean up the folder - when IL closes. Thus the protocol naturally takes care of "orphan" messages. This answers my question!

My whole post here is now a case of "talk to the duck." That's where you speak your problem out loud to an inanimate object. The upside is you don't waste anyone else's time serving as your sounding board. The downside is you don't get feedback, except what you hear yourself saying. The upside is you hear what the heck you're saying. The idea is to prevent a situation where you're in the middle of articulating your challenge, and you suddenly realize, Aha, that's what I'm doing wrong! Then you run away to test your idea, and the other person is left in the dust saying, "Happy to help, I guess?"

In this case, every subsequent round of "Testing shows" taught me something new about how the program operates, and how the spec has already solved my problem.

New Tobin Paradigm

Re: Obsolete unlisted message after deleting attachment

Unread post by New Tobin Paradigm » 2020-12-30, 00:07

When doing imap Interlink can only send and receive commands from the imap server no matter its local cache. If that is fucked then it is is fucked. So i guess the bottom line is because of your chosen configuration.. you simply fucked yourself and Interlink is doing what little it can to unfuck it. My suggestion is to try not to do that in the future.

Locked