Emails sent with Epyrus are scoring low according to SpamAssassin Topic is solved

[gabrgv]

I’m getting a lower score at https://www.mail-tester.com with emails sent using Epyrus. SpamAssassin gives a -1.596 score due to “FORGED_MUA_MOZILLA”, which is Epyrus related.

[Moonchild]

SpamAssassin needs to be taught that Epyrus exists so it no longer thinks it's a forged UA.
Not sure who's in charge of the rulesets there

[athenian200]

Well, I have no idea what this means or how to fix it, but I was able to reproduce your results easily.

I looked through the e-mail headers, and I'm wondering if this is the reason why it's being called a forged UA:

Code: Select all

Mozilla/5.0 (Windows NT 10.0; WOW64; rv:6.3) Goanna/20230916 Epyrus/2.1.0

The Mozilla/5.0 part may be being misinterpreted as forging Mozilla headers somehow?

Honestly, it looks like the way e-mail security is done these days is making the concept of an independent e-mail client like mine less and less viable... even less than an independent browser like Pale Moon. You have to be verified by several different organizations and basically have full-time staff to jump through all the hoops needed to not be seen as malware.

[athenian200]

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7411

So, apparently, if this is correct, any e-mail client with the word "Mozilla" in the user agent is now going to be regarded as impersonating Mozilla unless it resolves to a valid Mozilla message ID?

[Moonchild]

So, apparently, if this is correct, any e-mail client with the word "Mozilla" in the user agent is now going to be regarded as impersonating Mozilla unless it resolves to a valid Mozilla message ID?

Looks like it, because of:

Code: Select all

51 	header __MOZILLA_MUA            User-Agent =~ /^mozilla\b/i
52 	header __MOZILLA_MSGID          MESSAGEID =~ /^<[A-F\d]{8}\.[A-F1-9][A-F\d]{0,7}\@\S+>$/m
53 	meta FORGED_MUA_MOZILLA         (__MOZILLA_MUA && !__UNUSABLE_MSGID && !__MOZILLA_MSGID)

So if you have a UA with "mozilla" in it (anywhere, case-insensitive), it demands that you use the exact Mozilla-style messageID format or it complains about a forged Mozilla client.

I guess the simple workaround is to supply a non-mozilla user-agent by overriding it in preferences, but ultimately this rule should be adjusted to allow Epyrus.

[jobbautista9]

It look like removing the Mozilla/5.0 string is the correct solution for SpamAssassin, not just a workaround. The reply to athenian's comment in the bugzilla seems to imply that this Mozilla string was never supposed to be a thing and should be removed for newer clients.

I don't understand one thing though: if they think it's useless, why even bother checking if the Mozilla string is forged or not? Are there any significant amount of people still using the original Mozilla suite and not moving to SeaMonkey? Do people actually care if someone is writing from a "real" Mozilla client? I don't get how this should contribute to the spam score.

All-in-all I think if I'm going to ever self-host my email I'm gonna stay the hell out of SpamAssassin from now on for my spam filtering.

[Moonchild]

Pretty hostile response there, and...

There is no mechanism that uses any sort of 'compatibility string' in email.

...aside from SpamAssassin using it to disproportionately target mail clients that base themselves on Thunderbird...

NO ONE CARES.

...except them.

This feels like your typical FOSS purist passive-aggressive way of handling things they "would prefer not to see": add some poorly or undocumented "feature" that punishes anyone making software not exactly the way they would do it.

IMHO

If UXP needs to be patched to exclude UA altogether from the mailnews core, then I'm fine with that too. Patches welcome

[athenian200]

It look like removing the Mozilla/5.0 string is the correct solution for SpamAssassin, not just a workaround. The reply to athenian's comment in the bugzilla seems to imply that this Mozilla string was never supposed to be a thing and should be removed for newer clients.

Yeah, to be fair I probably should have opened a new bug rather than replying to that one, even though that was the one that arguably caused a regression, so it wasn't clear if I should reply on the bug that regressed this, or create a new one. But still, the info he provided was helpful... I checked and it does look like Outlook doesn't send this header. Meaning e-mail clients don't actually have to send a User-Agent header at all. Which is something I did not know, but how was I supposed to know that e-mail clients don't have to send User-Agent headers? A lot of them do.

Off-topic:
I gotta say though, this is actually how I am used to being treated by open source developers whenever I interact with them... they are super rude and technical, and if I'm lucky they give helpful information while dressing me down, and if I'm not lucky they just say nothing or give a one word reply before banning me from an IRC channel or mailing list failing to follow some arcane procedure that wasn't explained clearly. Might explain why I never learned to see that kind of behavior as a red flag... just saying. LOL.

But overall, I will have to say that I think this is kind of a stupid rule, because Mozilla is an open-source application and it's not exactly a secret how it creates the header. Anyone at all could get around this rule by simply making sure to format the header properly, so it would only ever flag people who either copy the Mozilla header incorrectly, or people who legitimately want to use a modified Mozilla-style header because their application was based on Mozilla code. So it would really only catch out very stupid spammers... it's not much of a trade secret, and copying that header really shouldn't be grounds for claiming that someone is impersonating Mozilla... but yeah, if it's true that we don't need it, then whatever. The less things named "Mozilla" I have attached to my client, the better.

[athenian200]

So, here is the issue where I researched this and found a relevant Mozilla bug where they seriously considered removing this header, just this year...

https://repo.palemoon.org/athenian200/epyrus/issues/72

And this is what I came up with as a preliminary patch.

https://repo.palemoon.org/athenian200/U ... 38073782f2

But then I realized that it's possible other MailNews devs may (or may not) want this for their applications, so it is probably worth having a discussion...

[Moonchild]

I realized that it's possible other MailNews devs may (or may not) want this for their applications, so it is probably worth having a discussion...

I don't think any other mail client sends User-agent strings at all in the message headers, so why not simply pref it and leave it out altogether, aligning with common practice where Mozilla is the odd one out?
Or rather, instead of what you're doing, we should use the standard practice of sending "Mailer:" or "X-Mailer:" headers with {name} {version}, if you want. Still should be preffed I think to allow people the choice whether they want to disclose this information or not...
EDIT: "X-Mailer:" seems to be the most common header used in the smattering of e-mails i had a quick look at, with pretty much arbitrary content ranging from "class SMTPmail" to "[name and revision][build date][debug version][profile][web client used to send mail]" -- either that or no mailer indicator at all (which would also be fine for any mail client building on UXP, I'm sure).

[athenian200]

I don't think any other mail client sends User-agent strings at all in the message headers, so why not simply pref it and leave it out altogether, aligning with common practice where Mozilla is the odd one out?
Or rather, instead of what you're doing, we should use the standard practice of sending "Mailer:" or "X-Mailer:" headers with {name} {version}, if you want. Still should be preffed I think to allow people the choice whether they want to disclose this information or not...
EDIT: "X-Mailer:" seems to be the most common header used in the smattering of e-mails i had a quick look at, with pretty much arbitrary content ranging from "class SMTPmail" to "[name and revision][build date][debug version][profile][web client used to send mail]" -- either that or no mailer indicator at all (which would also be fine for any mail client building on UXP, I'm sure).

That makes sense. I'm glad you were able to look into this and give me a better sense of what common practice is. If other mail clients did send that User-Agent header at one point (Mozilla in the bug I read seems convinced other mail clients do send it, but my own testing revealed modern Outlook doesn't), then they don't anymore.

It looks like the somewhat rude SpamAssassin developer was right about the header being totally unneeded, and RFC 7231 not being a thing with e-mail like Mozilla thought. So yeah, I've tested with this code removed, and e-mails go straight through, no complaining about it not meeting the MIME standard or anything. I guess I just assumed you wouldn't send a web browser-like User-Agent string in your e-mails unless it was required because... well, why else would you go to the trouble of including so much information about the OS and the fact that the software is compatible with Mozilla/5.0?

EDIT: I did more digging, and I found out what may have happened so many years ago that led to Mozilla being confused on this issue (because I can't just leave well enough alone):

https://bugzilla.mozilla.org/show_bug.cgi?id=65472
https://bugzilla.mozilla.org/show_bug.cgi?id=6929

Essentially, as a result of a bug at some point 20 years ago, Mozilla began displaying the X-Mailer header as User-Agent when other clients used that. Meaning that they would for a long time see a User-Agent from other e-mail clients that wasn't actually being sent, because they weren't checking the message source, and the software wasn't configured to display X-Mailer as its own field. The reason why they did that... is simple. They had a draft version of a protocol that specifically said X-Mailer was obsoleted in favor of User-Agent.

The things, upon closer inspection of that standard, it apparently became RFC 5536 in 2009:

https://www.rfc-editor.org/rfc/rfc5536#section-3.2.13

The thing about RFC 5536 is, it seems that as written, it applies to Netnews articles (which are a subset of e-mail), not explicitly to e-mail as a whole. Meaning that it's not clear whether the working group actually had the authority to obsolete the X-Mailer header for e-mail, because their scope was essentially USENET. Which means, if you're sending a message to a Newsgroup, you actually do need to send a User-Agent header... but this standard apparently never caught on with e-mail.

So basically, it seems that Mozilla adopted that standard thinking it was an official recommendation and that X-Mailer was obsolete. While everyone else basically just kept using X-Mailer, and to this day they are the only ones that implemented the recommendation, possibly because they also supported Newsgroups and therefore were affected by RFC 5536, when e-mail clients that don't work with Newsgroups wouldn't have that issue.

Later Mozilla contributors didn't understand where this came from, and confused this with RFC 7231, which is actually a totally different standard that also defines a User-Agent field. So Mozilla did base this on a standard, but their implementation was slightly wrong, they forgot why they did it, and no one else adopted it anyway.

[Moonchild]

Only Mozilla knows why they started including the user-agent string in mail headers. It's probably historical and no longer relevant either way.

I'd suggest either sending "X-Mailer: Epyrus" with an optional version, or nothing at all.
It's an optional header either way so no, it would not impact deliverability, at all, unless arbitrary filtering (like SA) makes a fuss about it.

[jb_wisemo]

Essentially, as a result of a bug at some point 20 years ago, Mozilla began displaying the X-Mailer header as User-Agent when other clients used that. Meaning that they would for a long time see a User-Agent from other e-mail clients that wasn't actually being sent, because they weren't checking the message source, and the software wasn't configured to display X-Mailer as its own field. The reason why they did that... is simple. They had a draft version of a protocol that specifically said X-Mailer was obsoleted in favor of User-Agent.

The things, upon closer inspection of that standard, it apparently became RFC 5536 in 2009:

https://www.rfc-editor.org/rfc/rfc5536#section-3.2.13

The thing about RFC 5536 is, it seems that as written, it applies to Netnews articles (which are a subset of e-mail), not explicitly to e-mail as a whole. Meaning that it's not clear whether the working group actually had the authority to obsolete the X-Mailer header for e-mail, because their scope was essentially USENET. Which means, if you're sending a message to a Newsgroup, you actually do need to send a User-Agent header... but this standard apparently never caught on with e-mail.

So basically, it seems that Mozilla adopted that standard thinking it was an official recommendation and that X-Mailer was obsolete. While everyone else basically just kept using X-Mailer, and to this day they are the only ones that implemented the recommendation, possibly because they also supported Newsgroups and therefore were affected by RFC 5536, when e-mail clients that don't work with Newsgroups wouldn't have that issue.

Later Mozilla contributors didn't understand where this came from, and confused this with RFC 7231, which is actually a totally different standard that also defines a User-Agent field. So Mozilla did base this on a standard, but their implementation was slightly wrong, they forgot why they did it, and no one else adopted it anyway.

So would the proper fix be to send/store the same data as "User-Agent" when posting to UseNet, but "X-Mailer" when not? (In particular changing this per transmission when sending to both address types, such as "To:comp.protocol.example; CC:example-group@ietf.org")

[athenian200]

So would the proper fix be to send/store the same data as "User-Agent" when posting to UseNet, but "X-Mailer" when not? (In particular changing this per transmission when sending to both address types, such as "To:comp.protocol.example; CC:example-group@ietf.org")

Yeah, that is ideally what I would do, but I am not sure how to separate this out so that it does something different when posting to a newsgroup than when sending an e-mail. The header seems to be defined in the same place for both.

It seems like Newsgroups do still accept X-Mailer even though according to RFC they should see it as deprecated, though. Probably because most of the older software used when they were popular supported X-Mailer unofficially and the popularity of it was mostly among people using it prior to the last RFC. This standard came out just as newsgroups were being replaced by online forums, so it wouldn't have gotten much attention.

[Pentium4User]

So would the proper fix be to send/store the same data as "User-Agent" when posting to UseNet, but "X-Mailer" when not? (In particular changing this per transmission when sending to both address types, such as "To:comp.protocol.example; CC:example-group@ietf.org")

Yeah, that is ideally what I would do, but I am not sure how to separate this out so that it does something different when posting to a newsgroup than when sending an e-mail. The header seems to be defined in the same place for both.

It seems like Newsgroups do still accept X-Mailer even though according to RFC they should see it as deprecated, though. Probably because most of the older software used when they were popular supported X-Mailer unofficially and the popularity of it was mostly among people using it prior to the last RFC. This standard came out just as newsgroups were being replaced by online forums, so it wouldn't have gotten much attention.

IRRC X- is all inofficial stuff, it is intended for use without standardisation.
User-Agent for Usenet is defined in the RFC.
ftp://ftp.rfc-editor.org/in-notes/rfc55 ... ion-3.2.13

Although I didn't find it for SMTP specification.

I also don't see a reason for SpamAssassin to check for User-Agent, because it is not mandatory to include it.

[athenian200]

So, I reactivated my old Eternal September account, and checked the source of the messages...

The majority of the clients there were using User-Agent or X-Newsreader, but it was hard to get a read on what the standard was because over half of the people still on Usenet, use Thunderbird or SeaMonkey to access Usenet.

Here's a sample of what the non-Mozilla clients on there were using:

Code: Select all

User-Agent: Pan/0.145 (Duplicitous mercenary valetism; d7e168a
 git.gnome.org/pan2)
 
User-Agent: ForteAgent/8.00.32.1272
 
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)

User-Agent: slrn/1.0.3 (Linux)

User-Agent: MacSOUP/2.8.6b1 (ed136d9b90) (Mac OS 10.14.6)

X-Newsreader: trn 4.0-test77 (Sep 1, 2010)

X-Newsreader: newsgroupstats.hk

X-Newsreader: Forte Agent 1.92/32.572

X-Newsreader: Microsoft Outlook Express 6.00.2900.5512
 

So it looks like User-Agent is the most common among up-to-date NNTP clients, but a ton of people still using USENET are using ancient clients designed for terminals on older Unix, or are even somehow using Microsoft Outlook Express, which I do remember supported newsgroups back in the day... and those clients all used X-Newsreader.

So Mozilla's code was mostly wrong, in that it sent User-Agent in e-mail, when it should have only sent it to NNTP. E-mail never switched from X-Mailer, while NNTP mostly switched from X-Newsgroup to User-Agent. Very weird, but that is what apparently happened.

The problem is that it seems like the composer code for both e-mails and newsgroup messages is shared in such a way as to make it not very obvious how to test whether a message will be sent to a newsgroup or an e-mail message recipient. I'm sure there has to be a way, because I think there are headers that get sent in Newsgroup messages only, but it isn't obvious at all looking at the code how you would do that.

[Pentium4User]

Outlook uses User-Agent.

There is a draft that suggested it for mail and it is BCP.
https://datatracker.ietf.org/doc/html/d ... r-agent-00

IANA only lists it for netnews (Usenet): http://www.iana.org/assignments/message ... ders.xhtml

In my opinion, spam filters must not check mails for "valid" or "common" User-Agent or X-Mailer headers, as they are not required.

[Moonchild]

In my opinion, spam filters must not check mails for "valid" or "common" User-Agent or X-Mailer headers, as they are not required.

It's low-hanging fruit for them if all mainstream clients send them and spambots often do not. The only reason this came up to begin with was because the most-deployed filter was causing issues, and Epyrus isn't "big enough" to consider. They want their "blocked" numbers to go up, after all, to satisfy people.

[Pentium4User]

The only reason this came up to begin with was because the most-deployed filter was causing issues, and Epyrus isn't "big enough" to consider. They want their "blocked" numbers to go up, after all, to satisfy people.

What does SpamAssassin say about that?

[Moonchild]

What does SpamAssassin say about that?

They said IIUC that nobody but official Mozilla software should send User-Agent in e-mails. That's why their filter tripped. Basically passing their responsibility off to us.