How do I disable compacting folders entirely? Topic is solved

[athenian200]

So, I have a little problem with Epyrus that I've been trying to figure out on my own for a while, by looking through the code (and the UI) and hoping I stumble across how it's structured... but I am just not understanding what I am seeing at all.

There's this dialog box that constantly pops up asking me if I want to compact folders to save 43 MB. I don't really want to do that, because it's such a ridiculously small amount of disk space and I have no idea if "compacting" things will have a negative impact on data integrity or not.

The checkbox on the dialog only offers to stop asking me, but if seems like if I uncheck it so that it stops asking, it will just compact folders by default. When I look at the available options in the UI, I see options to increase the threshold, but that's all that's listed.

There's also another checkbox under advanced options...

2023-12-09.png

This one is next to the threshold amount, but it's not clear what it does. Does it stop compacting altogether, or does it just not require a threshold to compact? The pref it controls is a bool called mail.prompt_purge_threshold which seems to imply that all it does is disable the threshold, not the compacting itself.

So there's a pref to control whether it asks you before compacting, and a pref that controls whether it uses a threshold to determine if it should compact, but nothing that unambiguously says it shouldn't automatically compact...

I normally would prefer to just play around with this to figure out how it works, but there's no easy way to do that in this case. It only appears on an actively used profile, and it wouldn't be clear to me if it was compacting in the background or not. Plus, if it did compact, there would be no easy way to get it back to the "uncompacted" state so I could keep testing if something I try doesn't work.

https://xref.palemoon.org/goanna-centra ... r.cpp#2005

I'm trying to parse this code out so I can understand it, but I just can't make sense of what it is doing or what order it's checking in... I feel like there's just too many branches and I can't keep it straight in my head. There's a setting that tells it to ask, there's a 5 minute timer, there's a maximum timer, there's a threshold amount, a threshold on or off, etc...

[Kris_88]

mail.prompt_purge_threshhold = false

https://xref.palemoon.org/goanna-centra ... r.cpp#1992
search: GetPromptPurgeThreshold

Some additional info: http://kb.mozillazine.org/Mail_and_news_settings

[Moonchild]

I have set mail.purge.ask to false so it "just does it" without asking. my threshold is set to 2048 and threshold_mb is 20 (default)
Pretty much hands-free. i never have to worry about compacting manually and it won't waste any disk space.

[athenian200]

mail.prompt_purge_threshhold = false

https://xref.palemoon.org/goanna-centra ... r.cpp#1992
search: GetPromptPurgeThreshold

Some additional info: http://kb.mozillazine.org/Mail_and_news_settings

That is some good info. I was thinking it might be that one, but it was a little counter-intuitive because it also controls whether it bothers to check for a threshold. So disabled means don't do auto-compact at all, and enabled means do it if the threshold is met...

Still, I am thinking that if compacting the folders automatically isn't causing problems, I should go ahead and let it compact automatically on my personal profile... and as for the Epyrus defaults, I wonder if it would be that bad an idea to make them less... umm... aggressive. Like, maybe it could check every 15 minutes and only activate if it's over 100MB... 20MB is like nothing on a modern hard drive.

I think the fact that it prompts made me worry about why it's prompting me... that's probably how I got it in my head that compacting must be risky. Probably I've been overthinking it this whole time.

[Moonchild]

Like, maybe it could check every 15 minutes and only activate if it's over 100MB... 20MB is like nothing on a modern hard drive.

I think those are sane defaults

compacting must be risky

It isn't. The warning I think is something primarily grandfathered in from when compacting would take considerable time on old, slow platter drives, during which you wouldn't really be able to use the mail client in any reasonable fashion because of bogging down with I/O

[athenian200]

Wound up looking on Bugzilla to see if the idea of increasing the threshold is crazy or if Mozilla considered it themselves after our fork point...

https://bugzilla.mozilla.org/show_bug.cgi?id=1462666

https://bugzilla.mozilla.org/show_bug.cgi?id=437657

Looks like it was increased to 200MB in TB 68, and on top of that, the original bug that set this threshold apparently pulled the 20MB amount out of thin air to begin with... 100MB was seriously considered even back when the decision was made to set it to 20MB in the first place. 20MB was a compromise because they were increasing it from 100KB at the time, and the change seemed extreme to people by that standard. So even if I set the default to 100MB in Epyrus, that's still more conservative than modern TB.

Also, learned that the reason that code is a bit confusing, is because the old code used KB instead of MB, and SeaMonkey kept using that for quite a while, so there's old migration paths and such in it, conversions from KB to MB, etc.

[The_Old_Mailman]

Privacy warning: on a IMAP server, if you don't regularly compact folders, it may happen that messages, that you don't see because deleted in your local client, are just marked as "to be expunged" on that server, even if you set the Epyrus option automatically to expunge deleted mails when closing the client.

This is particularly true for deleted messages in Trash and Junk folders. They are marked as "to be expunged" but never really expunged until those folders are compacted.

I've seen this same behavior in Thunderbird 115 too and it's rather disappointing because deleted mails were still present on the server and I was not aware of this behavior. It's simple to identify what folders contain messages marked as "to be expunged" that are not visible in Epyrus client by activating the folder size column in the folder pane. There, in apparently empty folders, you may see the size of messages to be compacted and therefore expunged.

Edit: I'm now experimenting the option that immediately executes the deletion, but it's rather risky if you do a mistake.

Edit2: I've just discovered that this behavior is true even for the draft folder! I suppose is true for whatever folder is not inbox, since you have to force Epyrus/Thunderbird to check all folders in order to have a correct count of messages on a IMAP server. In fact the default behavior of Thunderbird/Epyrus is checking Inbox only. Server side filters are ignored, by default. Old school users like me that use more a structure made by folders rather than labeling messages in the inbox may miss messages.

[athenian200]

Privacy warning: on a IMAP server, if you don't regularly compact folders, it may happen that messages, that you don't see because deleted in your local client, are just marked as "to be expunged" on that server, even if you set the Epyrus option automatically to expunge deleted mails when closing the client.

This is particularly true for deleted messages in Trash and Junk folders. They are marked as "to be expunged" but never really expunged until those folders are compacted.

I've seen this same behavior in Thunderbird 115 too and it's rather disappointing because deleted mails were still present on the server and I was not aware of this behavior. It's simple to identify what folders contain messages marked as "to be expunged" that are not visible in Epyrus client by activating the folder size column in the folder pane. There, in apparently empty folders, you may see the size of messages to be compacted and therefore expunged.

Edit: I'm now experimenting the option that immediately executes the deletion, but it's rather risky if you do a mistake.

Edit2: I've just discovered that this behavior is true even for the draft folder! I suppose is true for whatever folder is not inbox, since you have to force Epyrus/Thunderbird to check all folders in order to have a correct count of messages on a IMAP server. In fact the default behavior of Thunderbird/Epyrus is checking Inbox only. Server side filters are ignored, by default. Old school users like me that use more a structure made by folders rather than labeling messages in the inbox may miss messages.

Well, that's disappointing news. It sounds like I can't really do anything about that aggressive interval trying to compact folders every 5 minutes after all, otherwise it's a privacy risk? Once again, not obvious at all that this is the main function of that feature... it looked like it was more about saving disk space than about making sure deleted e-mails were actually deleted. Seriously... why can't it just delete things when it says delete, why does it have to work this way? This is dumb... I feel like I've started to understand why e-mail providers broke down and said to use their webmail interface instead of a local client. They probably got sick of being called about issues like this.

EDIT: Actually, it doesn't seem to matter one way or the other. The way this is configured out of the box even in Thunderbird, a junk folder cannot be purged more than once every 8 hours. So it checks every 5 minutes, but will not actually purge until 8 hours later...

[back2themoon]

Privacy warning: on a IMAP server, if you don't regularly compact folders, it may happen that messages...

This is a bit vague. How regularly to compact folders, and how often it may‏‏‎ happen?

I think athenian200 is correct in trying to establish a more sane default. Keep doing the compacting, just not five times a day.

I do remember a time where I was constantly asked to compact, so I increased the threshold gradually. I don't know what triggered the increased compacting requests. It seemed sudden and with no apparent reason. I'd blame OAuth2 for this, too.

[athenian200]

I think athenian200 is correct in trying to establish a more sane default. Keep doing the compacting, just not five times a day.

I do remember a time where I was constantly asked to compact, so I increased the threshold gradually. I don't know what triggered the increased compacting requests. It seemed sudden and with no apparent reason. I'd blame OAuth2 for this, too.

The 5 actually means it tries to check every 5 minutes to see if the amount that would be saved by compacting is above the threshold. There's another value that keeps it from actually doing the compacting more than 3 times a day, though.

I can pretty much confirm this, because it seems like for me Epyrus will nag me every 5 minutes if I tell it I don't want to compact folders... (because I never found that little option until now that lets you increase the threshold).

[The_Old_Mailman]

Well, that's disappointing news. It sounds like I can't really do anything about that aggressive interval trying to compact folders every 5 minutes after all, otherwise it's a privacy risk? Once again, not obvious at all that this is the main function of that feature... it looked like it was more about saving disk space than about making sure deleted e-mails were actually deleted. Seriously... why can't it just delete things when it says delete, why does it have to work this way? This is dumb... I feel like I've started to understand why e-mail providers broke down and said to use their webmail interface instead of a local client. They probably got sick of being called about issues like this.

EDIT: Actually, it doesn't seem to matter one way or the other. The way this is configured out of the box even in Thunderbird, a junk folder cannot be purged more than once every 8 hours. So it checks every 5 minutes, but will not actually purge until 8 hours later...

Is the 8 hour limit hardcoded?

It would a great addition to be able to set it via an option.

However, I think that this behavior is an historical heritage of the old Thunderbird when communications with IMAP servers were done on slow connections and more frequent access could cause overload of the server itself. But who knows...

[back2themoon]

The 5 actually means it tries to check every 5 minutes

I just put 5 as a random number, only meaning "way too often". 🙂

[athenian200]

Is the 8 hour limit hardcoded?

It would a great addition to be able to set it via an option.

However, I think that this behavior is an historical heritage of the old Thunderbird when communications with IMAP servers were done on slow connections and more frequent access could cause overload of the server itself. But who knows...

It's not hardcoded, actually, the 8 hour limit can be set by:

Code: Select all

// "mail.purge.min_delay", never purge a junk folder more than once every 480 minutes (60 mins/hour * 8 hours)
// "mail.purge.timer_interval", fire the purge timer every 5 minutes, starting 5 minutes after we load accounts
pref("mail.purge.min_delay", 480);
pref("mail.purge.timer_interval", 5);

So yeah, if you want it to work more often, you would need to reduce mail.purge.min_delay to something like, say, 240. As it stands, mail can only actually be purged three times a day (24 / 8 = 3), but it will check every 5 minutes. Which doesn't make a whole lot of sense...

[Moonchild]

You should set that interval to 30 minutes or so, at least. compacting should not be so ADHD-fueled. it's not important.
Heck, doing it at most once a day is already plenty, probably

[Bilbo47]

On a IMAP server, [before "Compaction" happens, deleted messages may silently persist] marked as "to be expunged" on the server [until] closing the client [when "Expungement" happens].

[Messages in] Trash and Junk folders [where more deletes happen] are marked as "to be expunged" but [not] expunged until the folders are compacted.

How does that make sense? Sure, "Expunge" is about telling the server to "commit" the message deletions, but "Compact" should be about the client performing its commits locally only. Like, in context of local accounts configured as 'mbox', where all messages in a folder are stored in one file. The local disk space formerly occupied by deleted messages is not recovered for other use until that folder/mbox file is "Compacted". (In the early days when disk space was at a premium and compacting was worth doing, it was the source of much file corruption, too often losing or scrambling many or all of the messages in the folder. Changing the mailstore file-scheme "Message Store Type" from all-in-one to one-for-one was seen as a boon, because it was no longer useful to run the buggy and/or failure-prone compaction code.) Does Compacting even apply to local accounts configured as MailDir, where each message is stored in its own file? I never see the Compacting prompt because my messages generally are stored on the server-side only, and not copied to local. (Account Settings => Synchronization & Storage => Message Synchronizing => [ ] Keep messages for this account on this computer) No local message files exist, neither per-message MailDir files nor per-folder mbox files, therefore no compacting can be done.

Also it seems silly to worry about non-privacy from not-yet-actually-deleted messages, considering that hosting servers take backups of all the mail all the time anyway. AND, within the past coupla years the good hosts have begun storing all mail as encrypted anyway; I assume decryption happens only via the account login, including password, before IMAP download.

[Moonchild]

"Expunge" is about telling the server to "commit" the message deletions, but "Compact" should be about the client performing its commits locally only.

This is correct. the two are unrelated.

[BenFenner]

A bit late to this discussion, but I always thought of "compacting" e-mail client folders like defragmenting a hard disc drive. I just assumed it was a very similar concept, due to the naming. Of course I'm sure I could go look it up...

[Moonchild]

I always thought of "compacting" e-mail client folders like defragmenting a hard disc drive

Well... yes and no. if you want to compare it, it would be like "defragmenting free space" as in the compacting moves the data together, eliminating gaps left by previously-deleted messages.

[BenFenner]

One could say defragmenting (or compacting) used space versus free space are two sides of the same coin. Or even exactly the same thing.
No?

[Moonchild]

Subtle difference
What compacting does, doesn't really describe either, but the focus is on the free space and not the data, hence me saying that. It's just a lot simpler than defragging a drive.
It's basically just packing all the messages to one side (the start of the message box files) and then at the end of the process cutting away whatever free space is left over by resizing the file down.