Why is Epyrus sometimes freezing mid-usage
Moderator: athenian200
Why is Epyrus sometimes freezing mid-usage
Back in the announcement thread I mentioned my hope that Epyrus would not randomly freeze like Interlink used to do. However, unfortunately this does happen with Epyrus 2.0.0 on Windows 8 (x86_64).
The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar. After a few minutes of using other programs for other tasks, Epyrus returns to polling for keyboard and mouse input (running the Windows Message Loop), and continues where it left off. But the problem may reappear some minutes later in the same run of Epyrus.
I have no known way of debugging, but I suspect that some background network operation is being run as blocking calls in the foreground (UI) thread, thus freezing the UI every time it happens during user input.
My configuration uses IMAP and NNTP with a lot of folders. IMAP is a site local dovecot server. The NNTP server is news.giganews.com.
The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar. After a few minutes of using other programs for other tasks, Epyrus returns to polling for keyboard and mouse input (running the Windows Message Loop), and continues where it left off. But the problem may reappear some minutes later in the same run of Epyrus.
I have no known way of debugging, but I suspect that some background network operation is being run as blocking calls in the foreground (UI) thread, thus freezing the UI every time it happens during user input.
My configuration uses IMAP and NNTP with a lot of folders. IMAP is a site local dovecot server. The NNTP server is news.giganews.com.
- back2themoon
- Moon Magic practitioner
- Posts: 2374
- Joined: 2012-08-19, 20:32
Re: Why is Epyrus sometimes freezing mid-usage
Have you checked with your security software and added exclusions for Epyrus? Since it's not a "popular" process, it's a prime target for aggressive behaviour from A/V's (Defender included).
Safe Mode / clean profile info: Help/Restart in Safe Mode
Information to include when asking for support - How to apply user agent overrides
How to auto-fill passwords
Windows 10 Pro x64
Information to include when asking for support - How to apply user agent overrides
How to auto-fill passwords
Windows 10 Pro x64
Re: Why is Epyrus sometimes freezing mid-usage
The slowdowns / temporary UI freezes (a few seconds at most) I have noticed from Epyrus usually happen immediately after deleting several messages. We would expect a background process to update the index file global-messages-db.sqlite at that time. However my file is 64MB, so I assume the delay has something to do with the large number (greater than one) of messages being updated all at the same time within the rather large index file.
Also have noticed a few general crashes per week, in context of keeping the program open all day, and closing it only when putting Windows to sleep for the night. These often happen when Epyrus does not have focus :/
Also have noticed a few general crashes per week, in context of keeping the program open all day, and closing it only when putting Windows to sleep for the night. These often happen when Epyrus does not have focus :/
Re: Why is Epyrus sometimes freezing mid-usage
I have Epyrus open pretty much 24/7 and have not experience any issues with it.
I also have huge mailboxes, and often delete many messages at once or in rapid succession. The only short (1 second or less) pauses I experience are expected when I e.g. move or archive a batch of messages. My GLODA message search index is 182 MB at the moment. No issues there...
If the search index is what you think is the problem then you could try rebuilding it from your mailboxes from a blank slate.
You may also want to check rebuilding folder indices if there's a specific mailbox/folder that causes issues.
Right-click a mailbox, select Properties, then click the "Repair folder" button
I also have huge mailboxes, and often delete many messages at once or in rapid succession. The only short (1 second or less) pauses I experience are expected when I e.g. move or archive a batch of messages. My GLODA message search index is 182 MB at the moment. No issues there...
If the search index is what you think is the problem then you could try rebuilding it from your mailboxes from a blank slate.
- Quit Epyrus.
- Locate the global-messages-db.sqlite file in your Epyrus user profile.
- Delete the global-messages-db.sqlite file.
- Start Epyrus. The re-indexing process will start automatically. Depending on the number of messages, it might take some time for the indexing to complete. You can continue to use Epyrus, but performance might be affected and search will return only partial results or even no results until the indexing is complete.
You may also want to check rebuilding folder indices if there's a specific mailbox/folder that causes issues.
Right-click a mailbox, select Properties, then click the "Repair folder" button
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Why is Epyrus sometimes freezing mid-usage
Just to correct some misunderstandings of my original report that have crept in:
1. My experience of this bug since the pre-Epyrus days provides no indication that it involves corrupt search indexes or recent mass deletions. Behavior feels more like it is caused by a scheduled activity such as autosave or "poll for new messages" . Often it happens halfway through typing a word in a new mail message, but it has also been seen when just navigating the message lists.
2. The connection to the mail server is plenty fast as it is often on the same gigabit LAN. However I cannot rule out that mail server disk access and/or backend authentication may take just enough time to delay some responses long enough to create the effect, in case Epyrus blocks the UI thread while waiting for related network responses.
3. As this doesn't happen every minute of every day, I cannot practically run Epyrus with a huge expanding debug log, nor find the exact moment in such a log where the problem is experienced. Maybe meaningful log messages to the (Windows) console from which Epyrus.exe was launched could allow me to switch windows and see what the last message on terminal during the Epyrus UI freeze is (This rules out a log that quickly flushes the relevant message out with a barrage of other messages).
4. A possible debug approach not involving me could be to log the time between Win32 message loop message retrievals and focus on whatever takes more than 10ms or 100ms (The user visible symptoms involve the Win32 mechanism to handle such a time exceeding 3000ms, but artificially using a lower limit would help identify potential hazardous code).
5. If antivirus is involved, it would (from the perspective of Epyrus) happen either during a network or file I/O request, which would be another thing to look at .
1. My experience of this bug since the pre-Epyrus days provides no indication that it involves corrupt search indexes or recent mass deletions. Behavior feels more like it is caused by a scheduled activity such as autosave or "poll for new messages" . Often it happens halfway through typing a word in a new mail message, but it has also been seen when just navigating the message lists.
2. The connection to the mail server is plenty fast as it is often on the same gigabit LAN. However I cannot rule out that mail server disk access and/or backend authentication may take just enough time to delay some responses long enough to create the effect, in case Epyrus blocks the UI thread while waiting for related network responses.
3. As this doesn't happen every minute of every day, I cannot practically run Epyrus with a huge expanding debug log, nor find the exact moment in such a log where the problem is experienced. Maybe meaningful log messages to the (Windows) console from which Epyrus.exe was launched could allow me to switch windows and see what the last message on terminal during the Epyrus UI freeze is (This rules out a log that quickly flushes the relevant message out with a barrage of other messages).
4. A possible debug approach not involving me could be to log the time between Win32 message loop message retrievals and focus on whatever takes more than 10ms or 100ms (The user visible symptoms involve the Win32 mechanism to handle such a time exceeding 3000ms, but artificially using a lower limit would help identify potential hazardous code).
5. If antivirus is involved, it would (from the perspective of Epyrus) happen either during a network or file I/O request, which would be another thing to look at .
- athenian200
- Contributing developer
- Posts: 1498
- Joined: 2018-10-28, 19:56
- Location: Georgia
Re: Why is Epyrus sometimes freezing mid-usage
Hi, I'm just noticing this report...
I've noticed the issue I think you're talking about on multiple UXP applications. Sometimes the UI just gets unresponsive for a few moments and won't register any keyboard input, and occasionally even long enough to falsely trigger the "Not Responding" thing. But then something gets unblocked and it's just fine again. I don't have any idea what causes it, but I know that the heavier the use of the application is, the more likely it is to happen. With browser applications, it seemed to happen when I had too many tabs open. With mail applications, it could happen when you have too many accounts listed.
If I understand correctly, it's not actually freezing, it's just getting so slow that Windows thinks it's not responding and acts like it is frozen, but eventually it recovers. The truth is, my use of Epyrus is not heavy enough to reproduce this issue and I really have no idea what to do about it...
I've noticed the issue I think you're talking about on multiple UXP applications. Sometimes the UI just gets unresponsive for a few moments and won't register any keyboard input, and occasionally even long enough to falsely trigger the "Not Responding" thing. But then something gets unblocked and it's just fine again. I don't have any idea what causes it, but I know that the heavier the use of the application is, the more likely it is to happen. With browser applications, it seemed to happen when I had too many tabs open. With mail applications, it could happen when you have too many accounts listed.
If I understand correctly, it's not actually freezing, it's just getting so slow that Windows thinks it's not responding and acts like it is frozen, but eventually it recovers. The truth is, my use of Epyrus is not heavy enough to reproduce this issue and I really have no idea what to do about it...
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind
Re: Why is Epyrus sometimes freezing mid-usage
When this happens try "Help" -> "Troubleshooting information" -> "about:memory" (in the "Application Basics" part, "Memory Use" line) -> "Measure and save" button. Post the resulting file.jb_wisemo wrote: ↑2022-11-28, 14:22The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar.
Re: Why is Epyrus sometimes freezing mid-usage
What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
- athenian200
- Contributing developer
- Posts: 1498
- Joined: 2018-10-28, 19:56
- Location: Georgia
Re: Why is Epyrus sometimes freezing mid-usage
The OP said he uses "a lot of folders," and presumably also has tons of accounts setup, from news servers to e-mail services. He didn't give a specific number, so I was imagining something in the neighborhood of 15-30 accounts. Which is more than I have actually tested Epyrus with because I don't really use multiple mail accounts.Moonchild wrote: ↑2023-09-09, 09:59What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind
Re: Why is Epyrus sometimes freezing mid-usage
My main account has about 30 folders (not counting the archived ones).
I'm not using it for news or rss though, so maybe it's related that if so.
I'm not using it for news or rss though, so maybe it's related that if so.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Why is Epyrus sometimes freezing mid-usage
athenian200 wrote: ↑2023-09-09, 15:11The OP said he uses "a lot of folders," and presumably also has tons of accounts setup, from news servers to e-mail services. He didn't give a specific number, so I was imagining something in the neighborhood of 15-30 accounts. Which is more than I have actually tested Epyrus with because I don't really use multiple mail accounts.Moonchild wrote: ↑2023-09-09, 09:59What is "too many"?
I currently use epyrus with 7 accounts in parallel and about 10G of stored mail in mailboxes, mix of IMAP and POP3. I'm not seeing any pauses in my use (and I leave the client up pretty much whenever I have the PC running which is most of the time).
When it happens, Epyrus is literally frozen and I can't do anything until I switch back to Epyrus later, and by then the internal state may have changed, but I'll try remembering that command next time anyway.Kris_88 wrote: ↑2023-09-08, 21:14When this happens try "Help" -> "Troubleshooting information" -> "about:memory" (in the "Application Basics" part, "Memory Use" line) -> "Measure and save" button. Post the resulting file.jb_wisemo wrote: ↑2022-11-28, 14:22The symptom is that while doing seemingly normal things like typing in the contents of an e-mail or navigating around the mail folders of various accounts, the UI suddenly stops responding to keystrokes, and after a few moments, Windows itself detects that Epyrus has stopped checking for new input and Appends "Not Responding" to the title bar.
As for heaviness of use my main IMAP account has over 1500 folders containing almost 500,000 messages, there's also an NNTP account and a WebCal account.
- athenian200
- Contributing developer
- Posts: 1498
- Joined: 2018-10-28, 19:56
- Location: Georgia
Re: Why is Epyrus sometimes freezing mid-usage
jb_wisemo wrote: ↑2023-10-01, 22:44When it happens, Epyrus is literally frozen and I can't do anything until I switch back to Epyrus later, and by then the internal state may have changed, but I'll try remembering that command next time anyway.
As for heaviness of use my main IMAP account has over 1500 folders containing almost 500,000 messages, there's also an NNTP account and a WebCal account.
Yeah... uh, you are definitely stress testing Epyrus more than I have so far. There could very well be some tweaks I need to deal with larger numbers of folders, and the NNTP code doesn't get a lot of updates because that code is pretty much removed upstream, so that might also need some work.
I mean, as long as you have plenty of RAM I would think it would be fine in theory, but then again parts of this application contain code from the 1990s and early 2000s, so there could be something in there gumming up the works and keeping it from working well with heavier use. I just honestly have not had to dive into the MailNews code the way I think I would have to in order to make it work better with more RAM and more cores...
"The Athenians, however, represent the unity of these opposites; in them, mind or spirit has emerged from the Theban subjectivity without losing itself in the Spartan objectivity of ethical life. With the Athenians, the rights of the State and of the individual found as perfect a union as was possible at all at the level of the Greek spirit." -- Hegel's philosophy of Mind
Re: Why is Epyrus sometimes freezing mid-usage
Because of the way IMAP works, having that many folders will inevitably hang up the client waiting for the network. Depending on your account settings, you may or may not have local copies of your IMAP mail. IMAP on accounts that have a ton of folders and messages also will impose pretty heavy load on the IMAP server, so that may also be a potential cause for delays, but I'm thinking actual application freezes are more likely due to synchronization and indexing tasks in the mail client itself. In Account Settings, under Synchronization and Storage, you can control this behaviour. You may want to limit synchronization of messages to a certain age or certain folders. You may also want to enable message archiving to reduce load.
NNTP could also be the culprit if you use it intensively, also because of networking and indexing/synchronization.
It's likely possible for someone with extensive knowledge of the protocols to troubleshoot this further and find out exactly where the freezes come from, but that isn't easily conveyed into instructions to follow for someone else. It'd need a "live" system displaying the problem to find the potential bottleneck, and since that's your private e-mail, that's likely not going to happen.
NNTP could also be the culprit if you use it intensively, also because of networking and indexing/synchronization.
It's likely possible for someone with extensive knowledge of the protocols to troubleshoot this further and find out exactly where the freezes come from, but that isn't easily conveyed into instructions to follow for someone else. It'd need a "live" system displaying the problem to find the potential bottleneck, and since that's your private e-mail, that's likely not going to happen.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Why is Epyrus sometimes freezing mid-usage
A number of those settings are named in the user interface to indicate they may work by causing deliberate loss or hiding of carefully grouped older messages, thus are too scary to enable or leave on. Most folders are set to be synchronized, but a few have been set to not synchronize at all if possible. Either way, the reported problem is that background mail tasks like synchronization, indexing and autosave need to be kept out of the thread that responds to keystrokes etc.Moonchild wrote: ↑2023-10-02, 00:22Because of the way IMAP works, having that many folders will inevitably hang up the client waiting for the network. Depending on your account settings, you may or may not have local copies of your IMAP mail. IMAP on accounts that have a ton of folders and messages also will impose pretty heavy load on the IMAP server, so that may also be a potential cause for delays, but I'm thinking actual application freezes are more likely due to synchronization and indexing tasks in the mail client itself. In Account Settings, under Synchronization and Storage, you can control this behaviour. You may want to limit synchronization of messages to a certain age or certain folders. You may also want to enable message archiving to reduce load.
Re: Why is Epyrus sometimes freezing mid-usage
Just let go of all those newsletters, man. My Gmail account turns 20 next year, I exclusively access it via IMAP and it has a total of 27k emails. Unlikely that you have 500k worth of personal email conversations. Newsletters are the easiest way to flood your mailbox even with a 15 GB free Gmail account, as I've seen with family members.
"One hosts to look them up, one DNS to find them and in the darkness BIND them."
Linux Mint 21 Xfce x64 on HP i5-5200 laptop, 12 GB RAM.
AutoPageColor|PermissionsPlus|PMPlayer|Pure URL|RecordRewind|TextFX
Linux Mint 21 Xfce x64 on HP i5-5200 laptop, 12 GB RAM.
AutoPageColor|PermissionsPlus|PMPlayer|Pure URL|RecordRewind|TextFX
Re: Why is Epyrus sometimes freezing mid-usage
Like I pointed out, you probably want to set up archiving in that case.
If you don't want to delete older messages, just archive them in a local folder structure in the client; having all of that remotely on the server is simply inevitably going to cause delays.
All of that is already asynchronous AFAIK. There's only so much you can do to prevent bottlenecks when dealing with very large volumes of messages.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
Re: Why is Epyrus sometimes freezing mid-usage
The "Archiving" options in the Epyrus/Thunderbird API are scarily vague as if someone at Mozilla was trying to automate the deletion of old mails in forgetful corporations, thus they are all off . However any folder with lots of mails has old messages manually moved to archival mail folders (still stored on the IMAP server), organized by year, year+month or particular subjects. This is one reason there are so many folders. The archives are on the server because it has better back up than the clients.
If the server takes a long time to respond to some automatic or background operation, freezing the client UI for the duration is the kind of bug I suspected.
Many of the mails aren't newsletters but status mails from servers, build systems etc. However there are also a few large mailing list subscriptions plus the UseNet groups (not included in the IMAP statistics).
If the server takes a long time to respond to some automatic or background operation, freezing the client UI for the duration is the kind of bug I suspected.
Many of the mails aren't newsletters but status mails from servers, build systems etc. However there are also a few large mailing list subscriptions plus the UseNet groups (not included in the IMAP statistics).
Re: Why is Epyrus sometimes freezing mid-usage
Not to tell you how to organize your workflow, but isn't that kind of archival best done with some form of cold storage? Is it really necessary to have instant access to status mails that are more than, say, a year old? Sounds really unnecessary to keep all status logs forever synced to clients.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite