ssl_error_rx_malformed_server_hello

[franstam]

hi moonchild,

i'm facing same error on another public site. i know this site would give a 404 also (thats expected, not the point here), but its reproducable.

https://kempte.ch/kemp-documentation

i also captured a tcpdump of the tls handshake and have it handy in case you need it?
i can also try to take a same tls handshake on the 32.0.1 version and compare too when i get home.

[Moonchild]

I actually get the error myself on that one (another bitly site, actually).
I'll investigate and see what's going on.
Please do send me the tcpdump so I can check.

[Moonchild]

I've verified this is indeed the TLS 1.3 downgrade sentinel throwing the error.
Flipping the option off in NSS allows for the connection to be established. I'll pref it for users who need this Issue #2180 (UXP). I've also contacted bitly about this issue as it seems at least some of their edge servers are misconfigured.

[franstam]

I actually get the error myself on that one (another bitly site, actually).
I'll investigate and see what's going on.
Please do send me the tcpdump so I can check.

whats the best way to send you the files?

[Moonchild]

I think I already figured it out but if you still want to send the tcpdump, just zip it up and attach to a PM

[Nun2Swoon]

I will forward this thread to OPNsense so that they may consider if the firewall handshaking is contributing to this problem.

[franstam]

I think I already figured it out but if you still want to send the tcpdump, just zip it up and attach to a PM

sent, thanks.

[Moonchild]

Thanks. It indeed shows the server hello was downgraded in the 32.1.0 handshake.
It does seem to be very fickle though, as I can't reproduce the error on the same system on a fresh build but the current release build continues to show the issue. it's totally weird, but a downgraded connection fail definitely happens here so yeah, my update should address that at least.
EDIT: The behaviour is actually as-expected when taking into account permanent redirects and various levels of caching. starting fresh though it does exactly what is expected, i.e.: failure to connect unless the downgrade sentinel is switched off.

[Anon658153]

Off-topic:

What are the other consequences of changing that setting, e.g. will I make other connections less secure by using it?

That depends on who you ask

In practice though, restricting yourself to TLS 1.2 isn't a problem. It is perfectly secure, just not the latest version of the protocol.

As someone a bit more paranoid and not at all surprised by $security_breach_of_the_week, disabling TLS 1.3 in the browser (by setting security.tls.version.max=3) is a footgun. TLS 1.2 may still be good enough for most use, but I wouldn't trust it to protect something like a connection to a financial institution over public wifi. https://www.cloudflare.com/learning/ssl/why-use-tls-1.3/ It's also possible that changing that setting will affect browser fingerprinting used by things like Cloudflare and Recaptcha, which could lead to loss of access to websites "protected" by them.

I've chosen to offtopic-tag all that because I believe the subject of the security of TLS 1.3 vs TLS 1.2 was brought up only because the suggested "fix" is not a fix and is actually a Very Bad Idea.

I can reproduce the issue with a basic Debian Linux + Apache server VM, which seems to rule out anything interfering with the connection. Based on my testing it seems to, at least partially, be an issue with PM 32.1.0 downgrading TLS 1.3 to 1.2 when it shouldn't. Disabling TLS 1.3 or turning off the downgrade protection are nasty workarounds that don't actually fix the real issue. They might appear to fix it, but I wouldn't do that on my daily-driver browser.

During testing I found that you can work around the issue by setting "security.tls.version.fallback-limit" to "4" in about:config. That will completely disable the ability for the browser to do any TLS fallback, thereby "correcting" the issue of the browser falling back when it shouldn't and then tripping the TLS 1.3 fallback detection.

For reference, the value "4" means "don't perform fallback to any TLS version less than 1.3" and is the same as the default value for that setting in recent versions of Firefox. "Fallback" is where the server advertises a version of TLS but the client can't connect at that version. With fallback-limit set to 4 the browser can still connect to TLS 1.2 servers as it isn't a fallback when the connection starts at a lower version. From a security perspective there's no valid reason for a fallback from TLS 1.3 to TLS 1.2, hence the fallback protection. The default value in PM 32.1.0 is 3, which means by default fallback from TLS 1.3 to TLS 1.2 is "allowed," but fallback from TLS 1.2 to anything lower is not.

I think there are multiple issues at play here. Moonchild's Wireshark screenshot shows the DOWNGRD tag coming from PM, but in my test the tag was sent by Apache. That screenshot also only shows a single TLS connection at TLS 1.2, whereas my test showed an initial connection at TLS 1.3, then a second connection at TLS 1.2. I've attached Wireshark screenshots and packet captures to this post.

Steps to reliably reproduce the issue and my test environment:

Install Debian 11 from netinstall iso. Configure with static IP and only install standard utilities and SSH.
Add sury.org Apache repo
Install Apache:
# apt install apache2 ssl-cert
Enable SSL and the default Apache SSL site:
# a2enmod ssl
# a2ensite default-ssl
# systemctl restart apache2

Load default site in PM 32.1.0 (https://vm.ip.address.here/) and get malformed hello error.
Load default site in PM 32.0.1 and get self-signed certificate warning. Accept cert and get default site.

My tests to confirm that the issue is an improper TLS downgrade in PM 32.1.0:

Each test was a modification of default settings for both Apache and Pale Moon. No TLS hardening was done to the server as it wasn't needed. All settings were reverted to defaults and both Apache and Pale Moon were restarted after each test.

• Setting security.tls.version.max=3 in PM 32.1.0 allows the browser to connect.
• Setting "SSLProtocol TLSv1.3" in Apache causes PM 32.1.0 to fail with the error SSL_ERROR_PROTOCOL_VERSION_ALERT. PM 32.0.1 is unaffected and connects normally. (I confirmed that the "SSLProtocol TLSv1.3" setting in Apache only allows TLS 1.3 by scanning it with the https://testssl.sh/ scanner.)
• Setting security.tls.version.fallback-limit=4 in PM 32.1.0 allows the browser to connect.
• Setting security.tls.version.min=4 in PM 32.1.0 allows the browser to connect.
• Setting security.tls.version.fallback-limit=4 in PM 32.1.0 and "SSLProtocol TLSv1.2" in Apache allows the browser to connect. (This test was mainly to check for issues connecting to TLS 1.2 servers with fallback disabled.)

I pulled a couple of packet captures from my OPNsense router during the testing and it looks like PM 32.1.0 with default settings is starting the connection at TLS 1.3, then abandoning that connection and initiating another connection using TLS 1.2. In the second connection, the server sends the DOWNGRD tag, then the browser responds to the server with an illegal parameter error and throws up the error message. I've attached Wireshark screenshots and the packet captures to this post.

Based on the results of those tests, it seems that PM 32.1.0 by default is triggering a TLS fallback where it isn't required, and at least in my case forcing a fallback where it's not even possible. (protocol_version_alert error) The tests indicate that PM 32.1.0 can connect using TLS 1.3, so it's not a lack of support. In my opinion a full fix would correct whatever is causing the browser to initiate a fallback, but from a security perspective simply disabling fallback entirely is acceptable. From a usability perspective there should be little or no impact from disabling fallback, as a fallback from TLS 1.3 to TLS 1.2 should cause the error by design, and a fallback to anything below TLS 1.2 is already not allowed by default.

For what it's worth, if it were me I'd revert the commit that adds the toggle to disable the fallback detection. I can understand the need for that option five years ago when TLS 1.3 had just been standardized and common support was lacking at best, but it's a major footgun now. IMO the fact that a Google search for the error message returns a bunch of now obsolete and arguably dangerous information only increases the caliber of said footgun.

Let me know if there's any additional testing you'd like me to perform.

[Moonchild]

Like I said before, it depends on who you ask.

TLS fallback existing isn't a bad thing. it actually allows you to connect with secure renegotiation to an earlier protocol version. Preventing fallback from TLS 1.3 altogether (even in a secure way) will cause issues elsewhere. The problem here is that a fallback is triggered in a way that is not secure, and that is a problem with server-side configurations. NSS has a sentinel for this which, with our update of the library, got enabled by default with no way for Pale Moon to control this (which will be addressed in the next point release).
Restricting the max TLS version to 1.2 is not a fix; it's a workaround in the interim. Your suggestion to prevent any fallback from TLS 1.3 is an alternative and more risky workaround that retains TLS 1.3 capabilities but may cause breakage on some sites, but also not a fix but a workaround. Even being able to switch off the sentinel as one can do in the next point release onwards is still a workaround, but with even smaller sec impact (will not cause breakage and will also preserve being able to fully use TLS 1.3). The only real solution and fix will have to be addressed server-side preventing this kind of fallback.

All of the workarounds offered though are valid and will not unnecessarily "break" connection security as TLS 1.2, the protocol falling back to, is perfectly fine to use from a practical security perspective. It contains all the features and controls for secure communication accepted by critical services like financial institutions and high-sec environments. Calling the workarounds a footgun is severely overstating things.

[Anon658153]

My explanation of fallback may not have been clear and I failed to site any authoritative sources with it. This post has authoritative citations.

I stand by my claim that changing the setting "security.tls.version.fallback-limit" to "4" will have no impact whatsoever on security and will not cause any interoperability issues. This is because there is no fallback from TLS 1.3 to TLS 1.2. Versions of TLS below 1.2 are deprecated. Therefore, there should no longer be fallback in TLS, at least by default, and the default for the fallback-limit setting should have been changed to "4" prior to this discussion. My authoritative sources for those statements are RFC 8446 and RFC 8996, along with RFC 2119 and RFC 8174 for clarity.

First off, please review RFC 2119 https://www.rfc-editor.org/rfc/rfc2119 and the short update to it RFC 8174 https://www.rfc-editor.org/rfc/rfc8174 to clear up any misunderstandings related to common key terms used in RFCs. I've copy/pasted them into the hide tags below:

RFC 2119

Network Working Group S. Bradner
Request for Comments: 2119 Harvard University
BCP: 14 March 1997
Category: Best Current Practice


Key words for use in RFCs to Indicate Requirement Levels

Status of this Memo

This document specifies an Internet Best Current Practices for the
Internet Community, and requests discussion and suggestions for
improvements. Distribution of this memo is unlimited.

Abstract

In many standards track documents several words are used to signify
the requirements in the specification. These words are often
capitalized. This document defines these words as they should be
interpreted in IETF documents. Authors who follow these guidelines
should incorporate this phrase near the beginning of their document:

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
RFC 2119.

Note that the force of these words is modified by the requirement
level of the document in which they are used.

1. MUST This word, or the terms "REQUIRED" or "SHALL", mean that the
definition is an absolute requirement of the specification.

2. MUST NOT This phrase, or the phrase "SHALL NOT", mean that the
definition is an absolute prohibition of the specification.

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

4. SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that
there may exist valid reasons in particular circumstances when the
particular behavior is acceptable or even useful, but the full
implications should be understood and the case carefully weighed
before implementing any behavior described with this label.





Bradner Best Current Practice [Page 1]

RFC 2119 RFC Key Words March 1997


5. MAY This word, or the adjective "OPTIONAL", mean that an item is
truly optional. One vendor may choose to include the item because a
particular marketplace requires it or because the vendor feels that
it enhances the product while another vendor may omit the same item.
An implementation which does not include a particular option MUST be
prepared to interoperate with another implementation which does
include the option, though perhaps with reduced functionality. In the
same vein an implementation which does include a particular option
MUST be prepared to interoperate with another implementation which
does not include the option (except, of course, for the feature the
option provides.)

6. Guidance in the use of these Imperatives

Imperatives of the type defined in this memo must be used with care
and sparingly. In particular, they MUST only be used where it is
actually required for interoperation or to limit behavior which has
potential for causing harm (e.g., limiting retransmisssions) For
example, they must not be used to try to impose a particular method
on implementors where the method is not required for
interoperability.

7. Security Considerations

These terms are frequently used to specify behavior with security
implications. The effects on security of not implementing a MUST or
SHOULD, or doing something the specification says MUST NOT or SHOULD
NOT be done may be very subtle. Document authors should take the time
to elaborate the security implications of not following
recommendations or requirements as most implementors will not have
had the benefit of the experience and discussion that produced the
specification.

8. Acknowledgments

The definitions of these terms are an amalgam of definitions taken
from a number of RFCs. In addition, suggestions have been
incorporated from a number of people including Robert Ullmann, Thomas
Narten, Neal McBurnett, and Robert Elz.












Bradner Best Current Practice [Page 2]

RFC 2119 RFC Key Words March 1997


9. Author's Address

Scott Bradner
Harvard University
1350 Mass. Ave.
Cambridge, MA 02138

phone - +1 617 495 3864

email - sob@harvard.edu









































Bradner Best Current Practice [Page 3]

RFC 8174

Internet Engineering Task Force (IETF) B. Leiba
Request for Comments: 8174 Huawei Technologies
BCP: 14 May 2017
Updates: 2119
Category: Best Current Practice
ISSN: 2070-1721


Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words

Abstract

RFC 2119 specifies common key words that may be used in protocol
specifications. This document aims to reduce the ambiguity by
clarifying that only UPPERCASE usage of the key words have the
defined special meanings.

Status of This Memo

This memo documents an Internet Best Current Practice.

This document is a product of the Internet Engineering Task Force
(IETF). It represents the consensus of the IETF community. It has
received public review and has been approved for publication by the
Internet Engineering Steering Group (IESG). Further information on
BCPs is available in Section 2 of RFC 7841.

Information about the current status of this document, any errata,
and how to provide feedback on it may be obtained at
http://www.rfc-editor.org/info/rfc8174.





















Leiba Best Current Practice [Page 1]

RFC 8174 RFC 2119 Clarification May 2017


Copyright Notice

Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.

Table of Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Clarifying Capitalization of Key Words . . . . . . . . . . . 3
3. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 4
4. Security Considerations . . . . . . . . . . . . . . . . . . . 4
5. Normative References . . . . . . . . . . . . . . . . . . . . 4
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 4

1. Introduction

RFC 2119 specifies common key words, such as "MUST", "SHOULD", and
"MAY", that may be used in protocol specifications. It says that the
key words "are often capitalized," which has caused confusion about
how to interpret non-capitalized words such as "must" and "should".

This document updates RFC 2119 by clarifying that only UPPERCASE
usage of the key words have the defined special meanings. This
document is part of BCP 14.

















Leiba Best Current Practice [Page 2]

RFC 8174 RFC 2119 Clarification May 2017


2. Clarifying Capitalization of Key Words

The following change is made to [RFC2119]:

=== OLD ===
In many standards track documents several words are used to signify
the requirements in the specification. These words are often
capitalized. This document defines these words as they should be
interpreted in IETF documents. Authors who follow these guidelines
should incorporate this phrase near the beginning of their document:

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119.


=== NEW ===
In many IETF documents, several words, when they are in all capitals
as shown below, are used to signify the requirements in the
specification. These capitalized words can bring significant clarity
and consistency to documents because their meanings are well defined.
This document defines how those words are interpreted in IETF
documents when the words are in all capitals.

o These words can be used as defined here, but using them is not
required. Specifically, normative text does not require the use
of these key words. They are used for clarity and consistency
when that is what's wanted, but a lot of normative text does not
use them and is still normative.

o The words have the meanings specified herein only when they are in
all capitals.

o When these words are not capitalized, they have their normal
English meanings and are not affected by this document.

Authors who follow these guidelines should incorporate this phrase
near the beginning of their document:

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
"MAY", and "OPTIONAL" in this document are to be interpreted as
described in BCP 14 [RFC2119] [RFC8174] when, and only when, they
appear in all capitals, as shown here.

=== END ===





Leiba Best Current Practice [Page 3]

RFC 8174 RFC 2119 Clarification May 2017


3. IANA Considerations

This document does not require any IANA actions.

4. Security Considerations

This document is purely procedural; there are no related security
considerations.

5. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>.

Author's Address

Barry Leiba
Huawei Technologies

Phone: +1 646 827 0648
Email: barryleiba@computer.org
URI: http://internetmessagingtechnology.org/



























Leiba Best Current Practice [Page 4]

I'll start with the easy one. TLS versions below 1.2 are deprecated. RFC 8996 is titled "Deprecating TLS 1.0 and TLS 1.1." https://www.rfc-editor.org/rfc/rfc8996 If the title is insufficient, the names for sections four and five, titled "Do Not Use TLS 1.0" and "Do Not Use TLS 1.1" respectively, should be sufficient.

This next part is longer as I need to clear up your misunderstandings regarding TLS 1.3. According to RFC 8446, titled "The Transport Layer Security (TLS) Protocol Version 1.3," https://www.rfc-editor.org/rfc/rfc8446 there is no renegotiation and no fallback with TLS 1.3. Page 27 contains the following:

Because TLS 1.3 forbids renegotiation, if a server has negotiated TLS 1.3 and receives a ClientHello at any other time, it MUST terminate the connection with an "unexpected_message" alert.

If a server established a TLS connection with a previous version of TLS and receives a TLS 1.3 ClientHello in a renegotiation, it MUST retain the previous protocol version. In particular, it MUST NOT negotiate TLS 1.3.

That should be sufficient to clear up the safety of my proposed preference change and proposed change to its default value. According to RFC 8446 there is no fallback from TLS 1.3 to a previous version. According to RFC 8996 TLS versions prior to 1.2 are deprecated, therefore there should not be a default ability for fallback from TLS 1.2 to a previous version. That means, logically, there should not be fallback at all at default settings. A connection is either TLS 1.2 or TLS 1.3. Once it starts at a certain level, it will stay at that level.

Next, your code commit https://repo.palemoon.org/MoonchildProd ... 33bcb1519c violates RFC 8446 when the new toggle is changed. Page 31 of RFC 8446 states the following:

TLS 1.3 has a downgrade protection mechanism embedded in the server's random value. TLS 1.3 servers which negotiate TLS 1.2 or below in response to a ClientHello MUST set the last 8 bytes of their Random
value specially in their ServerHello.

If negotiating TLS 1.2, TLS 1.3 servers MUST set the last 8 bytes of their Random value to the bytes:

44 4F 57 4E 47 52 44 01

If negotiating TLS 1.1 or below, TLS 1.3 servers MUST, and TLS 1.2 servers SHOULD, set the last 8 bytes of their ServerHello.Random value to the bytes:

44 4F 57 4E 47 52 44 00

TLS 1.3 clients receiving a ServerHello indicating TLS 1.2 or below MUST check that the last 8 bytes are not equal to either of these values. TLS 1.2 clients SHOULD also check that the last 8 bytes are not equal to the second value if the ServerHello indicates TLS 1.1 or below. If a match is found, the client MUST abort the handshake with an "illegal_parameter" alert.

I've added bold tags to the important sections. To be clear, all versions of Pale Moon prior to 32.1.0, going back at least to 28.14.2, are in violation of that section of the RFC. (28.14.2 is the oldest version for which I still have the Linux tarball.) PM 32.1.0 is the first version not in violation as far as I know.

Giving that code commit some thought, I can think of some EXTREMELY limited use cases for that toggle, all related to misconfigured or poorly-implemented TLS on local devices. It should not used to "regain access" to public websites such as the proposed use here.

I have no authoritative source for my claim that changing the preference value of "security.tls.version.downgrade-limit" to "4" will function as a short-term fix to the current TLS 1.3 downgrade bug in Pale Moon. It is easy enough to test yourself that it shouldn't need an authoritative source. The authoritative sources above are proof that it will not cause any harm, so changing that will, at worst, have no practical effect.

Finally, I'd like to challenge your claim that the issue is server-side and there's nothing to fix in Pale Moon. As I obviously have no authoritative sources for this, I'll ask some questions which rely on information that can be easily and independently verified:

If the problem is server-side, why is it that, out of Chromium 111, Firefox 111, Edge 111, Firefox 102.8, Opera 97, Pale Moon 3.1.0, and Pale Moon 3.0.1, all at new profile default settings, only Pale Moon 3.1.0 has the issue connecting to servers capable of both TLS 1.3 and TLS 1.2?

If the problem is server-side, why is it that, out of Chromium 111, Firefox 111, Edge 111, Firefox 102.8, Opera 97, Pale Moon 3.1.0, and Pale Moon 3.0.1, all at new profile default settings, only Pale Moon 3.1.0 and Pale Moon 3.0.1 are unable to connect at all to servers which are only capable of TLS 1.3?

If the problem is server-side, why is it that, out of Chromium 111, Firefox 111, Edge 111, Firefox 102.8, Opera 97, Pale Moon 3.1.0, and Pale Moon 3.0.1, all at new profile default settings, only Pale Moon 3.1.0 and Pale Moon 3.0.1 show both TLS 1.3 and TLS 1.2 in packet captures of connection attempts to TLS 1.3 capable servers?

If the problem is server-side, why is it that changing the fallback-limit setting from "3" to "4" verifiably fixes, across multiple servers, Pale Moon's issues connecting to TLS 1.3 serves using TLS 1.3?



I'm not here to assign blame. I couldn't care less who wrote the offending code or why. It doesn't matter. I would just like to see this fixed, and see workarounds pushed that don't involve disabling or crippling an important protocol. I would also like it made clear that any site which is triggering this bug is a site that Pale Moon could, at least since version 28.14.2, never establish a TLS 1.3 connection to and was always silently degrading down to TLS 1.2. Your proposed workaround will continue this behavior. A simple Wireshark capture will verify as much. This behavior is bad not only because it violates accepted interoperability standards, but also because TLS 1.2 is very much NOT "perfectly secure." My authoritative citations for that are the following Google searches: TLS FREAK, TLS LogJam, TLS BREACH, TLS CRIME, TLS TicketBleed, TLS LUCKY13, and TLS RC4.

For what it's worth, my offer to assist with testing still stands. One of my servers will already work as a test:

(I've added a spaces to try to cut down on web scraper bots and the like)
This link triggers the Pale Moon TLS fallback bug for me: https: //168.235.109.239/
This link does not trigger the bug: https: //www. watnet.org/

The first link, if you change a setting to get past the TLS handshake issue, will give you a cert error. If you add an exception it should give you a 403 page.
The second link should load a plain "website placeholder" page and should not present any errors.

Those are running on the same Apache service and, as far as I can tell, have identical TLS settings. I don't yet know why one works and the other doesn't, but I suspect it might have something to do with SNI as that seems to be the only difference.

[Moonchild]

there is no renegotiation and no fallback with TLS 1.3.

only once established

violates RFC 8446 when the new toggle is changed.

the toggle is a workaround, as I already explained multiple times. it is only there to provide users with an option to connect to misconfigured servers or middleware. This is no different than the option added for negotiation of the TLS 1.3 draft spec (which strictly speaking also violates the TLS 1.3 spec).

Pale Moon is a web client. The reality is that servers are misconfigured and devices can have flaws. It is definitely within scope to provide users with the options to still connect.
This is also why by default we still accept TLS 1.0 and 1.1 if that is what the server offers.

why is it that, out of Chromium 111, Firefox 111, Edge 111, Firefox 102.8, Opera 97, Pale Moon 3.1.0, and Pale Moon 3.0.1, all at new profile default settings, only Pale Moon 3.1.0 and Pale Moon 3.0.1 show both TLS 1.3 and TLS 1.2 in packet captures of connection attempts to TLS 1.3 capable servers?

Because fallback is disabled in Chromium and Firefox, period. With the exception of whitelists that aren't user-exposed, IIUC.

If the problem is server-side, why is it that changing the fallback-limit setting from "3" to "4" verifiably fixes, across multiple servers, Pale Moon's issues connecting to TLS 1.3 serves using TLS 1.3?

Because it disables fallback, period.

: TLS FREAK, TLS LogJam, TLS BREACH, TLS CRIME, TLS TicketBleed, TLS LUCKY13, and TLS RC4.

None of which are applicable in secure clients like Pale Moon. On top, a lot of those depend on legacy cipher suites which TLS 1.3 simply does not contain, period (it only offers very limited choice of ciphers). i.e.: in a proper implementation of TLS 1.2, it is perfectly secure. In fact, there are also known issues with TLS 1.3, in particular for example with the 0RTT and similar "must go faster" efforts that undermine a proper 3-way handshake for the sake of shaving a few ms off the negotiation.
See also: https://clienttest.ssllabs.com:8443/ssl ... lient.html with Pale Moon.

Now if you'll excuse me, I have other things that require my attention than discussing misconceptions about TLS in this thread.

[Claverhouse]

I get the (Error code: SSL_ERROR_RX_MALFORMED_SERVER_HELLO) bit on my bank ( NatWest in England, using Vodafone's own router/modem [ compulsory ] ); but just refreshing that page clears it.

Pale Moon 32.1.0 64bit

A lot better than 2 months back when --- once past that entry page, NatWest displayed large portions of code gibberish on specific portions of the bank statements...

[dapgo]

With Ublock disabled for the page, the error also appears in:
https://transvulcania.es

An error/warning also was shown on FIrefox and Chrome but there after confirming the exception the page loaded with a redirection to
https://transvulcania.utmb.world/es

[dapgo]

Regarding issue "(Error code: SSL_ERROR_RX_MALFORMED_SERVER_HELLO)"* with PM and Zscaler client connector

I realized that some pages were working fine on PM, such as google.com (Google Trust Services LLC), mozilla.org or stackoverflow(let´s encrypt)
whereas most of them were producing the error, including google.es

As firefox was able to load failing pages, I found out that key "security.enterprise_roots.enabled" is true in Firefox but false in PM.
I set to true and all failing pages began to load on PM.

*"Secure Connection Failed
An error occurred during a connection to yandex.com.
SSL received a malformed Server Hello handshake message.
(Error code: SSL_ERROR_RX_MALFORMED_SERVER_HELLO)

[jobbautista9]

I found out that key "security.enterprise_roots.enabled" is true in Firefox but false in PM.

That pref is not enabled by default in my Firefox 112 on Windows, so not sure why yours is different.

I tried enabling it anyway, and it didn't work for https://ircbot.comm-central.org:8080/, which currently has an expired TLS certificate. security.tls.hello_downgrade_check works, and is therefore still the best workaround in this case.

[Moonchild]

Enabling enterprise roots ( = reading the OS certificate store) comes with potential security drawbacks. More advanced malware will mess with the OS cert store to suppress errors for spoofing domains, something we're by default not vulnerable to because we use our own trust store that isn't that easily manipulated (there is no API to manipulate it outside of the browser, AFAIK)
Just so you are aware of that potential pitfall.
Even so it's strange that that would resolve this error since the error occurs before a TLS connection is established (that is: before any certificate check is performed).

[jobbautista9]

Anyway I've been modifying Firefox's about:config to match as closely to Pale Moon's defaults as possible, by setting the following:

"security.tls.enable_0rtt_data" to false
"security.tls.version.fallback-limit" to 3
"security.tls.version.min" to 1

And ensure security.tls.hello_downgrade_check is still set to true.

Strangely enough, Firefox doesn't give a SSL_ERROR_RX_MALFORMED_SERVER_HELLO for https://ircbot.comm-central.org:8080/ as I would've expected and went ahead with the error about the cert being expired. I tried setting "security.ssl.enable_tls13_compat_mode" to true in Pale Moon to see if it bypasses the server hello error, but it doesn't seem it's one of those middlebox sites... I wonder if we're missing something here that Mozilla has?

[Moonchild]

I wonder if we're missing something here that Mozilla has?

Unlikely. TLS sessions tend to be cached so once you've established a connection once, the hello tends to be different and may bypass the bad box issue. I had some trouble testing the workaround for the same reason on Bitly which was confusing me at first the same way

[blitzed]

Thank you for the troubleshooting everyone, and the work-around. I encountered the same gotcha, while attempting to access a DD-WRT router control panel via https://192.168.1.1/

Off-topic:

During testing I found that you can work around the issue by setting "security.tls.version.fallback-limit" to "4" in about:config. That will completely disable the ability for the browser to do any TLS fallback, thereby "correcting" the issue of the browser falling back when it shouldn't and then tripping the TLS 1.3 fallback detection.

For reference, the value "4" means "don't perform fallback to any TLS version less than 1.3" and is the same as the default value for that setting in recent versions of Firefox. "Fallback" is where the server advertises a version of TLS but the client can't connect at that version. With fallback-limit set to 4 the browser can still connect to TLS 1.2 servers as it isn't a fallback when the connection starts at a lower version. From a security perspective there's no valid reason for a fallback from TLS 1.3 to TLS 1.2, hence the fallback protection. The default value in PM 32.1.0 is 3, which means by default fallback from TLS 1.3 to TLS 1.2 is "allowed," but fallback from TLS 1.2 to anything lower is not.