Page 1 of 1

Why is Epyrus sometimes freezing mid-usage

Posted: 2022-11-28, 14:22
by jb_wisemo
Back in the announcement thread I mentioned my hope that Epyrus would not randomly freeze like Interlink used to do. However, unfortunately this does happen with Epyrus 2.0.0 on Windows 8 (x86_64).

The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar. After a few minutes of using other programs for other tasks, Epyrus returns to polling for keyboard and mouse input (running the Windows Message Loop), and continues where it left off. But the problem may reappear some minutes later in the same run of Epyrus.

I have no known way of debugging, but I suspect that some background network operation is being run as blocking calls in the foreground (UI) thread, thus freezing the UI every time it happens during user input.

My configuration uses IMAP and NNTP with a lot of folders. IMAP is a site local dovecot server. The NNTP server is news.giganews.com.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2022-12-03, 00:38
by back2themoon
Have you checked with your security software and added exclusions for Epyrus? Since it's not a "popular" process, it's a prime target for aggressive behaviour from A/V's (Defender included).

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2022-12-05, 20:24
by Bilbo47
The slowdowns / temporary UI freezes (a few seconds at most) I have noticed from Epyrus usually happen immediately after deleting several messages. We would expect a background process to update the index file global-messages-db.sqlite at that time. However my file is 64MB, so I assume the delay has something to do with the large number (greater than one) of messages being updated all at the same time within the rather large index file.

Also have noticed a few general crashes per week, in context of keeping the program open all day, and closing it only when putting Windows to sleep for the night. These often happen when Epyrus does not have focus :/

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2022-12-06, 01:50
by Moonchild
I have Epyrus open pretty much 24/7 and have not experience any issues with it.
I also have huge mailboxes, and often delete many messages at once or in rapid succession. The only short (1 second or less) pauses I experience are expected when I e.g. move or archive a batch of messages. My GLODA message search index is 182 MB at the moment. No issues there...

If the search index is what you think is the problem then you could try rebuilding it from your mailboxes from a blank slate.
  1. Quit Epyrus.
  2. Locate the global-messages-db.sqlite file in your Epyrus user profile.
  3. Delete the global-messages-db.sqlite file.
  4. Start Epyrus. The re-indexing process will start automatically. Depending on the number of messages, it might take some time for the indexing to complete. You can continue to use Epyrus, but performance might be affected and search will return only partial results or even no results until the indexing is complete.
The indexing progress can be monitored through the Tools > Activity Manager menu.

You may also want to check rebuilding folder indices if there's a specific mailbox/folder that causes issues.
Right-click a mailbox, select Properties, then click the "Repair folder" button

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-09-08, 18:50
by jb_wisemo
Just to correct some misunderstandings of my original report that have crept in:

1. My experience of this bug since the pre-Epyrus days provides no indication that it involves corrupt search indexes or recent mass deletions. Behavior feels more like it is caused by a scheduled activity such as autosave or "poll for new messages" . Often it happens halfway through typing a word in a new mail message, but it has also been seen when just navigating the message lists.

2. The connection to the mail server is plenty fast as it is often on the same gigabit LAN. However I cannot rule out that mail server disk access and/or backend authentication may take just enough time to delay some responses long enough to create the effect, in case Epyrus blocks the UI thread while waiting for related network responses.

3. As this doesn't happen every minute of every day, I cannot practically run Epyrus with a huge expanding debug log, nor find the exact moment in such a log where the problem is experienced. Maybe meaningful log messages to the (Windows) console from which Epyrus.exe was launched could allow me to switch windows and see what the last message on terminal during the Epyrus UI freeze is (This rules out a log that quickly flushes the relevant message out with a barrage of other messages).

4. A possible debug approach not involving me could be to log the time between Win32 message loop message retrievals and focus on whatever takes more than 10ms or 100ms (The user visible symptoms involve the Win32 mechanism to handle such a time exceeding 3000ms, but artificially using a lower limit would help identify potential hazardous code).

5. If antivirus is involved, it would (from the perspective of Epyrus) happen either during a network or file I/O request, which would be another thing to look at .

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-09-08, 21:12
by athenian200
Hi, I'm just noticing this report...

I've noticed the issue I think you're talking about on multiple UXP applications. Sometimes the UI just gets unresponsive for a few moments and won't register any keyboard input, and occasionally even long enough to falsely trigger the "Not Responding" thing. But then something gets unblocked and it's just fine again. I don't have any idea what causes it, but I know that the heavier the use of the application is, the more likely it is to happen. With browser applications, it seemed to happen when I had too many tabs open. With mail applications, it could happen when you have too many accounts listed.

If I understand correctly, it's not actually freezing, it's just getting so slow that Windows thinks it's not responding and acts like it is frozen, but eventually it recovers. The truth is, my use of Epyrus is not heavy enough to reproduce this issue and I really have no idea what to do about it...

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-09-08, 21:14
by Kris_88
jb_wisemo wrote:
2022-11-28, 14:22
The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar.
When this happens try "Help" -> "Troubleshooting information" -> "about:memory" (in the "Application Basics" part, "Memory Use" line) -> "Measure and save" button. Post the resulting file.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-09-09, 09:59
by Moonchild
athenian200 wrote:
2023-09-08, 21:12
it could happen when you have too many accounts listed.
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-09-09, 15:11
by athenian200
Moonchild wrote:
2023-09-09, 09:59
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
The OP said he uses "a lot of folders," and presumably also has tons of accounts setup, from news servers to e-mail services. He didn't give a specific number, so I was imagining something in the neighborhood of 15-30 accounts. Which is more than I have actually tested Epyrus with because I don't really use multiple mail accounts.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-09-09, 15:39
by Moonchild
My main account has about 30 folders (not counting the archived ones).
I'm not using it for news or rss though, so maybe it's related that if so.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-10-01, 22:44
by jb_wisemo
athenian200 wrote:
2023-09-09, 15:11
Moonchild wrote:
2023-09-09, 09:59
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
The OP said he uses "a lot of folders," and presumably also has tons of accounts setup, from news servers to e-mail services. He didn't give a specific number, so I was imagining something in the neighborhood of 15-30 accounts. Which is more than I have actually tested Epyrus with because I don't really use multiple mail accounts.
Kris_88 wrote:
2023-09-08, 21:14
jb_wisemo wrote:
2022-11-28, 14:22
The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar.
When this happens try "Help" -> "Troubleshooting information" -> "about:memory" (in the "Application Basics" part, "Memory Use" line) -> "Measure and save" button. Post the resulting file.
When it happens, Epyrus is literally frozen and I can't do anything until I switch back to Epyrus later, and by then the internal state may have changed, but I'll try remembering that command next time anyway.

As for heaviness of use my main IMAP account has over 1500 folders containing almost 500,000 messages, there's also an NNTP account and a WebCal account.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-10-01, 23:23
by athenian200
jb_wisemo wrote:
2023-10-01, 22:44
When it happens, Epyrus is literally frozen and I can't do anything until I switch back to Epyrus later, and by then the internal state may have changed, but I'll try remembering that command next time anyway.

As for heaviness of use my main IMAP account has over 1500 folders containing almost 500,000 messages, there's also an NNTP account and a WebCal account.
:shock:

Yeah... uh, you are definitely stress testing Epyrus more than I have so far. There could very well be some tweaks I need to deal with larger numbers of folders, and the NNTP code doesn't get a lot of updates because that code is pretty much removed upstream, so that might also need some work.

I mean, as long as you have plenty of RAM I would think it would be fine in theory, but then again parts of this application contain code from the 1990s and early 2000s, so there could be something in there gumming up the works and keeping it from working well with heavier use. I just honestly have not had to dive into the MailNews code the way I think I would have to in order to make it work better with more RAM and more cores...

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-10-02, 00:22
by Moonchild
Because of the way IMAP works, having that many folders will inevitably hang up the client waiting for the network. Depending on your account settings, you may or may not have local copies of your IMAP mail. IMAP on accounts that have a ton of folders and messages also will impose pretty heavy load on the IMAP server, so that may also be a potential cause for delays, but I'm thinking actual application freezes are more likely due to synchronization and indexing tasks in the mail client itself. In Account Settings, under Synchronization and Storage, you can control this behaviour. You may want to limit synchronization of messages to a certain age or certain folders. You may also want to enable message archiving to reduce load.

NNTP could also be the culprit if you use it intensively, also because of networking and indexing/synchronization.

It's likely possible for someone with extensive knowledge of the protocols to troubleshoot this further and find out exactly where the freezes come from, but that isn't easily conveyed into instructions to follow for someone else. It'd need a "live" system displaying the problem to find the potential bottleneck, and since that's your private e-mail, that's likely not going to happen.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-10-02, 00:47
by jb_wisemo
Moonchild wrote:
2023-10-02, 00:22
Because of the way IMAP works, having that many folders will inevitably hang up the client waiting for the network. Depending on your account settings, you may or may not have local copies of your IMAP mail. IMAP on accounts that have a ton of folders and messages also will impose pretty heavy load on the IMAP server, so that may also be a potential cause for delays, but I'm thinking actual application freezes are more likely due to synchronization and indexing tasks in the mail client itself. In Account Settings, under Synchronization and Storage, you can control this behaviour. You may want to limit synchronization of messages to a certain age or certain folders. You may also want to enable message archiving to reduce load.
A number of those settings are named in the user interface to indicate they may work by causing deliberate loss or hiding of carefully grouped older messages, thus are too scary to enable or leave on. Most folders are set to be synchronized, but a few have been set to not synchronize at all if possible. Either way, the reported problem is that background mail tasks like synchronization, indexing and autosave need to be kept out of the thread that responds to keystrokes etc.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-10-02, 04:44
by moonbat
jb_wisemo wrote:
2023-10-01, 22:44
1500 folders containing almost 500,000 messages
Just let go of all those newsletters, man. My Gmail account turns 20 next year, I exclusively access it via IMAP and it has a total of 27k emails. Unlikely that you have 500k worth of personal email conversations. Newsletters are the easiest way to flood your mailbox even with a 15 GB free Gmail account, as I've seen with family members.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-10-02, 10:27
by Moonchild
jb_wisemo wrote:
2023-10-02, 00:47
they may work by causing deliberate loss or hiding of carefully grouped older messages,
Like I pointed out, you probably want to set up archiving in that case.
If you don't want to delete older messages, just archive them in a local folder structure in the client; having all of that remotely on the server is simply inevitably going to cause delays.
jb_wisemo wrote:
2023-10-02, 00:47
Either way, the reported problem is that background mail tasks like synchronization, indexing and autosave need to be kept out of the thread that responds to keystrokes etc.
All of that is already asynchronous AFAIK. There's only so much you can do to prevent bottlenecks when dealing with very large volumes of messages.

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-11-01, 14:57
by jb_wisemo
The "Archiving" options in the Epyrus/Thunderbird API are scarily vague as if someone at Mozilla was trying to automate the deletion of old mails in forgetful corporations, thus they are all off . However any folder with lots of mails has old messages manually moved to archival mail folders (still stored on the IMAP server), organized by year, year+month or particular subjects. This is one reason there are so many folders. The archives are on the server because it has better back up than the clients.

If the server takes a long time to respond to some automatic or background operation, freezing the client UI for the duration is the kind of bug I suspected.

Many of the mails aren't newsletters but status mails from servers, build systems etc. However there are also a few large mailing list subscriptions plus the UseNet groups (not included in the IMAP statistics).

Re: Why is Epyrus sometimes freezing mid-usage

Posted: 2023-11-01, 15:05
by Moonchild
jb_wisemo wrote:
2023-11-01, 14:57
Many of the mails aren't newsletters but status mails from servers, build systems etc.
Not to tell you how to organize your workflow, but isn't that kind of archival best done with some form of cold storage? Is it really necessary to have instant access to status mails that are more than, say, a year old? Sounds really unnecessary to keep all status logs forever synced to clients.