Why is Epyrus sometimes freezing mid-usage

Board for discussions around the Epyrus mail and news client.

Moderator: athenian200

jb_wisemo
Moonbather
Moonbather
Posts: 56
Joined: 2016-01-27, 02:09

Why is Epyrus sometimes freezing mid-usage

Unread post by jb_wisemo » 2022-11-28, 14:22

Back in the announcement thread I mentioned my hope that Epyrus would not randomly freeze like Interlink used to do. However, unfortunately this does happen with Epyrus 2.0.0 on Windows 8 (x86_64).

The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar. After a few minutes of using other programs for other tasks, Epyrus returns to polling for keyboard and mouse input (running the Windows Message Loop), and continues where it left off. But the problem may reappear some minutes later in the same run of Epyrus.

I have no known way of debugging, but I suspect that some background network operation is being run as blocking calls in the foreground (UI) thread, thus freezing the UI every time it happens during user input.

My configuration uses IMAP and NNTP with a lot of folders. IMAP is a site local dovecot server. The NNTP server is news.giganews.com.

User avatar
back2themoon
Moon Magic practitioner
Moon Magic practitioner
Posts: 2358
Joined: 2012-08-19, 20:32

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by back2themoon » 2022-12-03, 00:38

Have you checked with your security software and added exclusions for Epyrus? Since it's not a "popular" process, it's a prime target for aggressive behaviour from A/V's (Defender included).

User avatar
Bilbo47
Fanatic
Fanatic
Posts: 232
Joined: 2017-11-18, 04:24

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Bilbo47 » 2022-12-05, 20:24

The slowdowns / temporary UI freezes (a few seconds at most) I have noticed from Epyrus usually happen immediately after deleting several messages. We would expect a background process to update the index file global-messages-db.sqlite at that time. However my file is 64MB, so I assume the delay has something to do with the large number (greater than one) of messages being updated all at the same time within the rather large index file.

Also have noticed a few general crashes per week, in context of keeping the program open all day, and closing it only when putting Windows to sleep for the night. These often happen when Epyrus does not have focus :/

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35404
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Moonchild » 2022-12-06, 01:50

I have Epyrus open pretty much 24/7 and have not experience any issues with it.
I also have huge mailboxes, and often delete many messages at once or in rapid succession. The only short (1 second or less) pauses I experience are expected when I e.g. move or archive a batch of messages. My GLODA message search index is 182 MB at the moment. No issues there...

If the search index is what you think is the problem then you could try rebuilding it from your mailboxes from a blank slate.
  1. Quit Epyrus.
  2. Locate the global-messages-db.sqlite file in your Epyrus user profile.
  3. Delete the global-messages-db.sqlite file.
  4. Start Epyrus. The re-indexing process will start automatically. Depending on the number of messages, it might take some time for the indexing to complete. You can continue to use Epyrus, but performance might be affected and search will return only partial results or even no results until the indexing is complete.
The indexing progress can be monitored through the Tools > Activity Manager menu.

You may also want to check rebuilding folder indices if there's a specific mailbox/folder that causes issues.
Right-click a mailbox, select Properties, then click the "Repair folder" button
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

jb_wisemo
Moonbather
Moonbather
Posts: 56
Joined: 2016-01-27, 02:09

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by jb_wisemo » 2023-09-08, 18:50

Just to correct some misunderstandings of my original report that have crept in:

1. My experience of this bug since the pre-Epyrus days provides no indication that it involves corrupt search indexes or recent mass deletions. Behavior feels more like it is caused by a scheduled activity such as autosave or "poll for new messages" . Often it happens halfway through typing a word in a new mail message, but it has also been seen when just navigating the message lists.

2. The connection to the mail server is plenty fast as it is often on the same gigabit LAN. However I cannot rule out that mail server disk access and/or backend authentication may take just enough time to delay some responses long enough to create the effect, in case Epyrus blocks the UI thread while waiting for related network responses.

3. As this doesn't happen every minute of every day, I cannot practically run Epyrus with a huge expanding debug log, nor find the exact moment in such a log where the problem is experienced. Maybe meaningful log messages to the (Windows) console from which Epyrus.exe was launched could allow me to switch windows and see what the last message on terminal during the Epyrus UI freeze is (This rules out a log that quickly flushes the relevant message out with a barrage of other messages).

4. A possible debug approach not involving me could be to log the time between Win32 message loop message retrievals and focus on whatever takes more than 10ms or 100ms (The user visible symptoms involve the Win32 mechanism to handle such a time exceeding 3000ms, but artificially using a lower limit would help identify potential hazardous code).

5. If antivirus is involved, it would (from the perspective of Epyrus) happen either during a network or file I/O request, which would be another thing to look at .

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1481
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by athenian200 » 2023-09-08, 21:12

Hi, I'm just noticing this report...

I've noticed the issue I think you're talking about on multiple UXP applications. Sometimes the UI just gets unresponsive for a few moments and won't register any keyboard input, and occasionally even long enough to falsely trigger the "Not Responding" thing. But then something gets unblocked and it's just fine again. I don't have any idea what causes it, but I know that the heavier the use of the application is, the more likely it is to happen. With browser applications, it seemed to happen when I had too many tabs open. With mail applications, it could happen when you have too many accounts listed.

If I understand correctly, it's not actually freezing, it's just getting so slow that Windows thinks it's not responding and acts like it is frozen, but eventually it recovers. The truth is, my use of Epyrus is not heavy enough to reproduce this issue and I really have no idea what to do about it...
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Kris_88
Keeps coming back
Keeps coming back
Posts: 932
Joined: 2021-01-26, 11:18

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Kris_88 » 2023-09-08, 21:14

jb_wisemo wrote:
2022-11-28, 14:22
The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar.
When this happens try "Help" -> "Troubleshooting information" -> "about:memory" (in the "Application Basics" part, "Memory Use" line) -> "Measure and save" button. Post the resulting file.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35404
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Moonchild » 2023-09-09, 09:59

athenian200 wrote:
2023-09-08, 21:12
it could happen when you have too many accounts listed.
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1481
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by athenian200 » 2023-09-09, 15:11

Moonchild wrote:
2023-09-09, 09:59
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
The OP said he uses "a lot of folders," and presumably also has tons of accounts setup, from news servers to e-mail services. He didn't give a specific number, so I was imagining something in the neighborhood of 15-30 accounts. Which is more than I have actually tested Epyrus with because I don't really use multiple mail accounts.
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35404
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Moonchild » 2023-09-09, 15:39

My main account has about 30 folders (not counting the archived ones).
I'm not using it for news or rss though, so maybe it's related that if so.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

jb_wisemo
Moonbather
Moonbather
Posts: 56
Joined: 2016-01-27, 02:09

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by jb_wisemo » 2023-10-01, 22:44

athenian200 wrote:
2023-09-09, 15:11
Moonchild wrote:
2023-09-09, 09:59
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
The OP said he uses "a lot of folders," and presumably also has tons of accounts setup, from news servers to e-mail services. He didn't give a specific number, so I was imagining something in the neighborhood of 15-30 accounts. Which is more than I have actually tested Epyrus with because I don't really use multiple mail accounts.
Kris_88 wrote:
2023-09-08, 21:14
jb_wisemo wrote:
2022-11-28, 14:22
The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar.
When this happens try "Help" -> "Troubleshooting information" -> "about:memory" (in the "Application Basics" part, "Memory Use" line) -> "Measure and save" button. Post the resulting file.
When it happens, Epyrus is literally frozen and I can't do anything until I switch back to Epyrus later, and by then the internal state may have changed, but I'll try remembering that command next time anyway.

As for heaviness of use my main IMAP account has over 1500 folders containing almost 500,000 messages, there's also an NNTP account and a WebCal account.

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1481
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by athenian200 » 2023-10-01, 23:23

jb_wisemo wrote:
2023-10-01, 22:44
When it happens, Epyrus is literally frozen and I can't do anything until I switch back to Epyrus later, and by then the internal state may have changed, but I'll try remembering that command next time anyway.

As for heaviness of use my main IMAP account has over 1500 folders containing almost 500,000 messages, there's also an NNTP account and a WebCal account.
:shock:

Yeah... uh, you are definitely stress testing Epyrus more than I have so far. There could very well be some tweaks I need to deal with larger numbers of folders, and the NNTP code doesn't get a lot of updates because that code is pretty much removed upstream, so that might also need some work.

I mean, as long as you have plenty of RAM I would think it would be fine in theory, but then again parts of this application contain code from the 1990s and early 2000s, so there could be something in there gumming up the works and keeping it from working well with heavier use. I just honestly have not had to dive into the MailNews code the way I think I would have to in order to make it work better with more RAM and more cores...
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35404
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Moonchild » 2023-10-02, 00:22

Because of the way IMAP works, having that many folders will inevitably hang up the client waiting for the network. Depending on your account settings, you may or may not have local copies of your IMAP mail. IMAP on accounts that have a ton of folders and messages also will impose pretty heavy load on the IMAP server, so that may also be a potential cause for delays, but I'm thinking actual application freezes are more likely due to synchronization and indexing tasks in the mail client itself. In Account Settings, under Synchronization and Storage, you can control this behaviour. You may want to limit synchronization of messages to a certain age or certain folders. You may also want to enable message archiving to reduce load.

NNTP could also be the culprit if you use it intensively, also because of networking and indexing/synchronization.

It's likely possible for someone with extensive knowledge of the protocols to troubleshoot this further and find out exactly where the freezes come from, but that isn't easily conveyed into instructions to follow for someone else. It'd need a "live" system displaying the problem to find the potential bottleneck, and since that's your private e-mail, that's likely not going to happen.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

jb_wisemo
Moonbather
Moonbather
Posts: 56
Joined: 2016-01-27, 02:09

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by jb_wisemo » 2023-10-02, 00:47

Moonchild wrote:
2023-10-02, 00:22
Because of the way IMAP works, having that many folders will inevitably hang up the client waiting for the network. Depending on your account settings, you may or may not have local copies of your IMAP mail. IMAP on accounts that have a ton of folders and messages also will impose pretty heavy load on the IMAP server, so that may also be a potential cause for delays, but I'm thinking actual application freezes are more likely due to synchronization and indexing tasks in the mail client itself. In Account Settings, under Synchronization and Storage, you can control this behaviour. You may want to limit synchronization of messages to a certain age or certain folders. You may also want to enable message archiving to reduce load.
A number of those settings are named in the user interface to indicate they may work by causing deliberate loss or hiding of carefully grouped older messages, thus are too scary to enable or leave on. Most folders are set to be synchronized, but a few have been set to not synchronize at all if possible. Either way, the reported problem is that background mail tasks like synchronization, indexing and autosave need to be kept out of the thread that responds to keystrokes etc.

User avatar
moonbat
Knows the dark side
Knows the dark side
Posts: 4942
Joined: 2015-12-09, 15:45
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by moonbat » 2023-10-02, 04:44

jb_wisemo wrote:
2023-10-01, 22:44
1500 folders containing almost 500,000 messages
Just let go of all those newsletters, man. My Gmail account turns 20 next year, I exclusively access it via IMAP and it has a total of 27k emails. Unlikely that you have 500k worth of personal email conversations. Newsletters are the easiest way to flood your mailbox even with a 15 GB free Gmail account, as I've seen with family members.
"One hosts to look them up, one DNS to find them and in the darkness BIND them."

Image
Linux Mint 21 Xfce x64 on HP i5-5200 laptop, 12 GB RAM.
AutoPageColor|PermissionsPlus|PMPlayer|Pure URL|RecordRewind|TextFX

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35404
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Moonchild » 2023-10-02, 10:27

jb_wisemo wrote:
2023-10-02, 00:47
they may work by causing deliberate loss or hiding of carefully grouped older messages,
Like I pointed out, you probably want to set up archiving in that case.
If you don't want to delete older messages, just archive them in a local folder structure in the client; having all of that remotely on the server is simply inevitably going to cause delays.
jb_wisemo wrote:
2023-10-02, 00:47
Either way, the reported problem is that background mail tasks like synchronization, indexing and autosave need to be kept out of the thread that responds to keystrokes etc.
All of that is already asynchronous AFAIK. There's only so much you can do to prevent bottlenecks when dealing with very large volumes of messages.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

jb_wisemo
Moonbather
Moonbather
Posts: 56
Joined: 2016-01-27, 02:09

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by jb_wisemo » 2023-11-01, 14:57

The "Archiving" options in the Epyrus/Thunderbird API are scarily vague as if someone at Mozilla was trying to automate the deletion of old mails in forgetful corporations, thus they are all off . However any folder with lots of mails has old messages manually moved to archival mail folders (still stored on the IMAP server), organized by year, year+month or particular subjects. This is one reason there are so many folders. The archives are on the server because it has better back up than the clients.

If the server takes a long time to respond to some automatic or background operation, freezing the client UI for the duration is the kind of bug I suspected.

Many of the mails aren't newsletters but status mails from servers, build systems etc. However there are also a few large mailing list subscriptions plus the UseNet groups (not included in the IMAP statistics).

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35404
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Why is Epyrus sometimes freezing mid-usage

Unread post by Moonchild » 2023-11-01, 15:05

jb_wisemo wrote:
2023-11-01, 14:57
Many of the mails aren't newsletters but status mails from servers, build systems etc.
Not to tell you how to organize your workflow, but isn't that kind of archival best done with some form of cold storage? Is it really necessary to have instant access to status mails that are more than, say, a year old? Sounds really unnecessary to keep all status logs forever synced to clients.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Post Reply