IMAP, location copies, and syncronization Topic is solved

[Navigator]

With synchronization turned on, what happens to the local copy of a message within IMAP folders if the message disappears from the server?

For example if someone gained access to my email account and deleted all my server-side messages, will Epyrus keep my messages when I connect and sync or will it start purging my local copies because they are not on the server?

[Moonchild]

IMAP is a server-side storage mechanism. What is stored locally is at most there for local caching/performance. So if the server messages are gone, then any locally cached copies of those messages will also be gone after synchronization. It's one of the main advantages AND disadvantages of using IMAP.

Bottom line: if you use IMAP, make sure your account is properly secured for access.

[Navigator]

So there is no way to tell Epyrus not to purge these messages? I hoped that like Pale Moon it would let me have it my way in most things, rather than forcing my hand.

Since I have backups I should be protected against a complete deletion as I would see that all my messages were gone and restore from backup. However if my messages are insidiously disappearing would I even have notice of this? Will Epyrus log that XX messages were deleted server-side and purged from the local offline mirror?

[athenian200]

I just tested this to find out, and it seems like it depends on where the e-mail is stored. If it's stored in a folder that's being synced with IMAP with a remote folder, then yes, it would be removed. However, if you copy the message to a local folder that isn't being synced, that copy would not be deleted.

That is to say, if you delete a message from your inbox, that message will be gone from your Inbox when it's synced with the IMAP server. But if you copy your inbox to a local folder that's not synced with the IMAP server, deleting the e-mail from the server only affects the copy in the Inbox, not the copy you stored locally in a local folder.

So my proposal is that if you don't quite trust your IMAP server to hold onto all your e-mails, copy the ones you want to keep to a local folder where they will be safe from being deleted on the next synchronization. The IMAP protocol works by simply keeping a series of local folders in sync with remote ones, so unfortunately deletion is synchronized as well. Some people actually prefer this feature to be the way it is, because it saves them from having to delete their e-mails twice, once on the server, and once locally. The thing is, if it didn't delete the local copy, it would be sent back to the IMAP server on the next synchronization.

The best way around it might be to just create a rule that forwards all your mail to a local folder as soon as you receive it. Just go to Message Filters and create a rule that copies any message you get to a local folder the moment you receive new mail. Not ideal, I know, because you're stuck with two copies of the message (one in your official "inbox" that's synced up, and one in your local folder that's getting all the copies of those messages), but it frees you to delete your messages from the server without worry if that's something you wanted to do.

Also worth noting that this works the other way around... if you copy anything from a local folder or other account to your inbox for some reason, that message will be automatically copied to every device you own that is synced to that account, it won't just stay locally, even if you never explicitly sent it to yourself.

[Moonchild]

I hoped that like Pale Moon it would let me have it my way in most things, rather than forcing my hand.

It's not "forcing your hand". IMAP is, in a way, one of the earliest forms of cloud computing that were created. Your mail client displays what is on the server because it is a server-based protocol. Using locally stored data is literally not what IMAP does; the protocol has no mechanisms for it because that is not its purpose.
Having it "your way" in terms of customization does not mean that protocols can suddenly be told to do things they were not designed for

However if my messages are insidiously disappearing would I even have notice of this? Will Epyrus log that XX messages were deleted server-side and purged from the local offline mirror?

No, and that is by design! IMAP does not "mirror" anything. What is stored in the client locally is only a temporary cache for performance reasons, and is not even guaranteed to exist, depending on which client you're using. IMAP clients display data as-it-is on the server, and they are designed to access one account from multiple locations (concurrently, too). If that server data changes, then it will also change in the client. Its whole purpose is to keep an as recent and up-to-date state of the server displayed in the client as possible. Warnings when messages change or are removed on the server are not desirable. If it would do that, then every other client would spam those messages whenever you send, move or remove anything on one connected client.

[athenian200]

Yeah. I should clarify that there is a difference between the folders associated with an account, like the one labeled "Inbox" and a folder labeled as being under the "Local Folders," which is a mechanism we have in place by default. If you copy every e-mail you receive to one of those folders the moment you receive it, like with a rule or something, then that e-mail would not be deleted. It's the same principle as if you copied a file from OneDrive or Google Drive to a folder on your hard drive, and then deleted it from OneDrive... the copy would still be there in the local folder, but it wouldn't appear on any device that was synced to OneDrive. The "Local Folders" mechanism is like a folder on your hard drive, and the folders listed under the account name are like OneDrive or Google Drive... that's not a perfect analogy, but that's sort of how it works.

Basically, to get around this, you need to copy your e-mails to a folder that's outside of this whole "caching/mirroring" mechanism. There's nothing in IMAP that stops a client from copying your e-mails to a separate folder that isn't mirrored on the server, but then you would need two folders... one folder that's just a synced copy of what's on the server and works as IMAP expects, and another that the server doesn't know about, which just contains redundant local copies of every e-mail you ever received. There's pretty much no other way to do it, and Epyrus already has a way to do that with filtering rules if you really want to, AFAIK.

I tested, and this does seem to work... even if I delete an e-mail I've never read before using a different e-mail client than Epyrus, and then go to read it in the local folder, I can still read it this way. Here's an example:

messagefilter.png

So, I haven't actually read every e-mail Duolingo ever sent me. I decided to try using this filter rule to copy all my Duolingo messages into the folder "test." Then, I went into another e-mail client and deleted every e-mail I ever got from Duolingo. As expected, all the Duolingo e-mails were gone from my Inbox on Epyrus, too. However, when I went to the "test" folder, all the e-mails from Duolingo that had been copied by the filter rule previously were still there. If you want to do this with all your messages, then just use "Match all messages." Make sure it runs every time you get new mail, and then you wind up with a local copy of every e-mail Epyrus actually gets that isn't kept in sync with the IMAP server.

For me personally though, I've always had the opposite problem... IMAP was very good at keeping my e-mails for years, whereas back when I used POP, I would lose all my e-mails every few years when I had to reinstall my operating system, or at best have to rely on random backups that were hard to load (often only designed to be imported by old versions of Outlook), that were often just not worth the trouble to access one or two e-mails. But once I started using IMAP, I found that it was fairly easy to keep up with every e-mail ever as long as I have enough storage space on the server. That is to say, I lost more e-mails relying on local storage, than I ever did relying on IMAP... if that's not the case for you, it sounds like something is wrong.

[Navigator]

Thank you both for helping me to better understand this. Before Epyrus I used POP3 exclusively but I thought I would try what I perceived as the modern way for convenient access from multiple machines. I realize now that IMAP is not intended for what I want to achieve but I still don't understand the technical limitations.

Since the offline-cache is not a bona fide local copy I will need to maintain a separate local folder tree as I did under POP3. This means that if I use the offline-cache feature there will actually be two copies of any message on my local machine, one in the cache and one in the Local Folders. This is space-inefficient. I will also need to delete any message twice to actually delete it, which is time-inefficient.

With my ancient POP3 client there was a feature like "delete email from POP3 when emptied from Trash." Usually the message was already gone as I also used the separate "delete from server after XX days" function. If I go back to POP3 and don't use a time-delete feature will this leave a persistent copy on the server? Could that persistent server-copy be moved to a server-subfolder somehow, and still have a "delete from server when emptied from Trash" functionality work?

If I keep using IMAP, turn off the offline cache, and add a filter to copy all messages to a Local Folders location, is there a way to periodically synchronize the IMAP server to my local master, turning the IMAP server into the mirror rather than the other way around?

[athenian200]

Since the offline-cache is not a bona fide local copy I will need to maintain a separate local folder tree as I did under POP3.

Correct. You would effectively be creating a "dumb" POP3-style local folder tree in addition to your IMAP one, so that the IMAP protocol doesn't have anything to say about it.

This means that if I use the offline-cache feature there will actually be two copies of any message on my local machine, one in the cache and one in the Local Folders. This is space-inefficient. I will also need to delete any message twice to actually delete it, which is time-inefficient.

Also correct. Probably why MC didn't offer this solution... the only obvious way around the limitations of IMAP would be a literal waste of time and space. But still, the point was more that you have the freedom to do it, not that it's a very practical thing to do.

With my ancient POP3 client there was a feature like "delete email from POP3 when emptied from Trash." Usually the message was already gone as I also used the separate "delete from server after XX days" function. If I go back to POP3 and don't use a time-delete feature will this leave a persistent copy on the server? Could that persistent server-copy be moved to a server-subfolder somehow, and still have a "delete from server when emptied from Trash" functionality work?

Really depends on how the POP3 server is configured. Some of them will delete your mail the moment you download it, meaning it's entirely on you to keep a copy of your message once it is downloaded, and also making it hard to get a copy of the e-mail on another client if you use multiple e-mail clients. Delete from server when emptied from trash is basically the POP3 equivalent of IMAP's default behavior... your e-mails are gone from the server once you delete it in any e-mail client. But with POP3, previously downloaded messages stay downloaded by OTHER clients that downloaded them before they were deleted from the server. So if you delete from trash on one client, it's gone from the server, but another e-mail client that already got the message would have it stay put. Which is annoying if you actually wanted to delete your e-mail from everywhere, right?

So with IMAP, any client that previously download the e-mails will see them gone because they were deleted from the server. It works this way precisely because of what you said above... deleting your e-mail twice is not very efficient, and most people like deleting their e-mail once and having it apply everywhere. Not great if you don't trust your IMAP provider not to delete e-mails on you behind your back, but pretty good if you can be confident in the server operator.

If I keep using IMAP, turn off the offline cache, and add a filter to copy all messages to a Local Folders location, is there a way to periodically synchronize the IMAP server to my local master, turning the IMAP server into the mirror rather than the other way around?

I don't know what happens if you turn off the offline cache, but I know that the way Epyrus is configured out of the box, if you copy anything INTO your local inbox using a filter rule, it will wind up on the IMAP server. I actually use this to read RSS feeds in Outlook, which longer has native support for them. I just have Epyrus copy RSS feeds into my Inbox, and read them on another e-mail client. It's funny seeing the Mozilla-formatted HTML and improper e-mail headers just blindly copied over to the server with forum usernames as the "from" sender, and then reading them in Outlook. Which means, yes, the "mirroring" works both ways, at least in my experience. Copy pretty much anything at all into your Inbox, and it will wind up in the inbox of the IMAP server, and hence will be readable on any other e-mail client you use. That's an example of me abusing the way IMAP works, but hopefully it helps you understand the issues involved.

To give you an idea of why this counter-intuitive idea works, think about the fact that you have an outbox or sent folder. When you send an e-mail, a copy of something you created locally is saved to the outbox and ends up on the server, right? Same applies to every other folder. So as things stand, it works both ways in Epyrus. Anything you put in your local mirrored copy that wasn't there before gets send to the IMAP server, and anything that was deleted from the server's copy gets deleted from your local copy. This is the reason why I said that if e-mail were not deleted from your inbox when it was detected that the server had deleted it, that e-mail would just be copied back to the server. Hence why the feature can't quite work the way you want... failing to delete those e-mails from the synched folder would result in them being restored by Epyrus back to the remote IMAP server, as if they had been placed there again, and Epyrus would effectively prevent you from ever being able to delete anything from the IMAP server unless you do it through Epyrus, and would just undo the deletion every time.

Does that make sense? Not sure if it helps you, but I'm explaining it to the best of my understanding.

[Navigator]

... failing to delete those e-mails from the synched folder would result in them being restored by Epyrus back to the remote IMAP server, as if they had been placed there again, and Epyrus would effectively prevent you from ever being able to delete anything from the IMAP server unless you do it through Epyrus, and would just undo the deletion every time.

Does that make sense? Not sure if it helps you, but I'm explaining it to the best of my understanding.

Actually that is exactly the behavior I would love to achieve! No I do not trust remote servers: too many data breaches, too may sudden closures or paywalls (FotoTime, Photobucket, etc.), too much censorship for me to feel any other way. But apparently my goal is simply too peculiar to expect support. Thanks again for the help and information nevertheless.

[athenian200]

Actually that is exactly the behavior I would love to achieve! No I do not trust remote servers: too many data breaches, too may sudden closures or paywalls (FotoTime, Photobucket, etc.), too much censorship for me to feel any other way. But apparently my goal is simply too peculiar to expect support. Thanks again for the help and information nevertheless.

Interesting. So, that really is what you want? I assumed I must be missing something, but now I get what you were trying to say. Well, that sounds like it could in theory be achieved... I just tried copying one of the "deleted" e-mails back to my inbox from the test folder just to verify this works, and sure enough the e-mail was restored on every device. Once again bypassing the way IMAP normally works. Presumably then, all I would have to do is simply find the code that honors the part of the IMAP protocol related to deletion of local copies of e-mail when it doesn't find it remotely, remove that, while leaving the part that copies anything it finds locally to the remote server in place. It would technically be breaking the IMAP protocol, though, and effectively turn Epyrus into an anti-deletion guard program that monitors the IMAP server for deletions that were not done through Epyrus itself, and restores anything in its cache to that server if it's missing, rather than a normal e-mail client that propagates/honors remote deletions normally.

Overall though, it sounds like if your e-mail server does support POP3 access, that would be better for your peace of mind. POP3 doesn't trust the remote server, puts more power in the hands of the client, and will let you keep all your local e-mails until you delete them. If I were in the situation you're in, that's what I would do. Just use POP3 and maybe use tricks like setting it up so that when I delete local e-mail, it's deleted remotely too. That's about as good as you can do without trusting the remote server.

I mean, as you can see, we would pretty much have to set up a parallel POP3-style structure locally so as to take away the remote server's power. I think my previously mentioned approach of a redundant folder structure would make sense for an IMAP-only e-mail server that doesn't allow POP3 access, but if yours does, then there's not really a good reason to use IMAP given your concerns. All the possible solutions involve inventing something that is halfway between IMAP and POP3 in one way or another, basically.

That said, I do wonder if someone could write an extension that keeps track of e-mail deletions on IMAP sync in some sort of log file without making too much noise.

[ag0044]

I'm not sure if this helps, nor if it is even relevant, but there is an option in Epyrus that seems to allow you to store your emails on the local machine.

From the menu, "Tools", "Account Settings". Expand the IMAP account if necessary, and select "Synchronization & Storage". At the top of the resulting window is a tick-box for "Keep messages for this account on this computer".

Further down that window there are further options.

I haven't played with the various settings to see what would happen. But if the OP sets (that is, ticks) the first option to store their messages locally, it may fulfill their need?

[Navigator]

Interesting. So, that really is what you want? I assumed I must be missing something, but now I get what you were trying to say. Well, that sounds like it could in theory be achieved... I just tried copying one of the "deleted" e-mails back to my inbox from the test folder just to verify this works, and sure enough the e-mail was restored on every device. Once again bypassing the way IMAP normally works. Presumably then, all I would have to do is simply find the code that honors the part of the IMAP protocol related to deletion of local copies of e-mail when it doesn't find it remotely, remove that, while leaving the part that copies anything it finds locally to the remote server in place. It would technically be breaking the IMAP protocol, though, and effectively turn Epyrus into an anti-deletion guard program that monitors the IMAP server for deletions that were not done through Epyrus itself, and restores anything in its cache to that server if it's missing, rather than a normal e-mail client that propagates/honors remote deletions normally.

As I said I don't expect you to pursue this. I thought it might be a simple flag like /PURGE on robocopy, but as Moonchild made clear it goes against the intent of IMAP. It is tantalizing as the local cache/copy functionality seems to work perfectly since I can read all my old messages even when offline.


Without giving up on IMAP I think I can achieve (most of) what I want through Filters. I don't need a complicated folder structure on the server side; the default Inbox, Sent, and Archives will do and still let me access my older messages from other devices when I want to. My plan is to create two Filters:

Apply Filter When: After sending
Match All Messages
Copy Message To: Local Folders/Sent

Apply Filter When: Archiving
Match All Messages
Copy Message To: Local Folders/Received

I believe this will keep a Local copy of anything I Send and anything I Archive, without making a Local copy of trivial incoming mail that I want to delete rather than archiving. Messages in Inbox will only exist in the offline cache and on the server. I can then deselect the Archives tree in Items for Offline Use to prevent double local copies. Is my understanding correct?


If I copy messages from different accounts into the same Local folder is there a robust way to separate these again that does not rely on the "To:" header? Eudora used to apply an "X-Persona:" header that could be used this way and that could not easily be spoofed.

[Navigator]

I'm not sure if this helps, nor if it is even relevant, but there is an option in Epyrus that seems to allow you to store your emails on the local machine.

I am making use of this option, and it appears to be robust in that I can access all my messages when offline. However the issue and the point of this thread is that these messages are not secure from remote deletion, accidental or otherwise.

[Bilbo47]

The option "Keep messages for this account on this computer" does not do exactly what its name says.

When this option is Off, message headers are cached to the local folder. Headers and other metadata are stored in the <Foldername>.msf files.

When this option is On, message headers and also message bodies are cached to the local folder. In addition to the MSF file, the bodies are stored in the <Foldername> file (no extension). In case the separate option for message storage scheme is set to file-per-message instead of file-per-folder, then each message body lives in its own local file in that cache folder. I believe this feature exists to allow reading messages while in Offline Mode.

As already noted, none of these options affect whether/how IMAP performs its cache synchronization. Message creation and deletion are propagated from server to client(s) regardless. The exception is the Local Folders account, which is a local-only / non-synced mailstore "database".

Here's an idea: back up the Local Folders account to the cloud, using your favorite automatic file-based cloud-sync scheme. This would not involve the IMAP server at all. I suppose you could also sync the cloud copy on another device, so the mail client on that device could have a separate auto-synced copy of the Local Folders account.

Edit: please excuse my imprecision. By "the local folder" I mean the locally-cached copy of the (server-side) subfolder inside an IMAP account. This is in contrast to the one non-IMAP account named "Local Folders", which can contain its own subfolders.