Data breach post-mortem

About this bulletin board and the Pale Moon website

Moderators: satrow, Lootyhoof, FranklinDM

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 24831
Joined: 2011-08-28, 17:27
Location: 58°2'16"N 14°58'31"E
Contact:

Data breach post-mortem

Unread post by Moonchild » 2019-07-10, 14:34

There has been a data breach on the archive server (archive.palemoon.org) where an attempt was made to sabotage our project by infecting all archived executables on the server with a trojan/virus dropper. This post-mortem report is posted to provide full transparency to our community as to what happened (as far can be gathered -- see below), which files were affected, what you can do to verify your downloads and what will be done to prevent such breaches in the future.

What happened?

A malicious party gained access to the at the time Windows-based archive server (archive.palemoon.org) which we've been renting from Frantech/BuyVM, and ran a script to selectively infect all archived Pale Moon .exe files stored on it (installers and portable self-extracting archives) with a variant of Win32/ClipBanker.DY (ESET designation). Running these infected executables will drop a trojan/backdoor on your system that would potentially allow further compromise to it.

The moment this was reported to me on 2019-07-09, I shut down access to the archive server to prevent any potential further spread of infected binaries and to start an investigation.

When did this happen?

According to the date/time stamps of the infected files, this happened on 27 December 2017 at around 15:30. It is possible that these date/time stamps were forged.

UPDATE: After getting some more feedback from users, the breach seems to have occurred much more recently (which makes sense, considering it would be really odd that this would have gone unnoticed for such a long time...). Estimate now is somewhere between April and June 2019.

How did this happen?

Our data on this is limited, because in a later incident (likely by the same party or one other with similar access) on 2019-05-26 the archive server was rendered completely inoperable to the point of having widespread data corruption and being unable to boot or retrieve data from it. Unfortunately that also means that system logs providing exact details of the breach were lost at that time.

After becoming inoperable, I set up the archive server again on a different O.S. (moved from Windows to CentOS, and changed access from FTP to HTTP as a result considering Linux FTP can't be easily set up the same way and this server is purely a convenience service for users).

Judging by the modified time stamps, the files were infected in rapid succession, increasing the file size by about 3 MB of malicious payload. They were infected locally on the system, most likely with a script performing direct file manipulations. The infected files were not uploaded remotely in their infected state.

By process of elimination of potential access, and user accounts that could gain access in the manner required to perform this infection, it's clear that this infection happened through:
  • Local access to the system (physical access), OR
  • Access to the VM from a different VM on the same node (insufficient separation), OR
  • Access to the VM from a different VM on the same local subnet through and insecure/hijacked remote desktop session (insufficient separation), OR
  • Access to the VM file system via administrative access to the O.S. (potentially after brute-forcing credentials) over the network (e.g. SAMBA/WFS) (insufficient VMnet separation/not blocking FS ports in the node/DC), OR
  • Access to the VM through remote access via the VM control panel (insecure control panel of the VM provider), OR
  • An issue with the provided Windows Server image (which was pre-activated/volume licensed by the VM provider).
I've ruled out remote FTP access, remote RDP access and execution of insecure software on the VM as potential breach points considering this access was at all times limited to myself only and locked down by IP and with secure, unique password protection. Remote security of the Windows O.S. was solid.

What was affected?

This affected all archived executables (installers and portable exes) of Pale Moon 27.6.2 and below. Archived versions of Basilisk on the same storage server, although some would have already been present at that time, were not affected or targeted. Only files on the archive server were infected. This never affected any of the main distribution channels of Pale Moon, and considering archived versions would only be updated when the next release cycle would happen, at no time any current versions, no matter where they were retrieved from, would be infected.
Of note: only the .exe files on the server at the top level were affected. Files inside the archives (extract-able with 7-zip from the installers/portable versions or files inside the zip archives) were not modified.

Unfortunately, after the incident that rendered the server inoperable, the files transferred to the new system were taken from a backup made earlier that was already in an infected state due to the passage of time that this breach has gone undetected, so the infected binaries were carried over to the new (CentOS) solution.

How do I verify my downloaded files are clean?

If you never downloaded from archive.palemoon.org, you are almost certainly in the clear. To be sure, I do recommend using at least one of the steps below.

Most of the versions of Pale Moon come with accompanying .sig files (pgp signatures) which you can use to verify that the files are not tampered with or changed in any way.
Apart from the period where code-signing was not used due to unavailability at reasonable cost for Open Source development, binaries have also been code-signed; which can be checked through a right-click -> properties -> tab "Digital Signatures". If this tab is missing, then the binary is either not signed or has been modified from its original.
Later versions of the archived executables also come with a SHA256 has in the accompanying file "hashes.txt". You can verify the integrity that way, too.
For a (somewhat) complete list of SHA256 hashes of clean files, see https://pastebin.com/Lp27meQe
Additionally, the infection is known to all major antivirus vendors and you can scan your downloads/system with your preferred mainstream antivirus scanner to verify the installers are clean.

If you have inadvertently run an infected installer or portable self-extractor, I suggest you do a full scan and clean of your system with reputable antivirus software to clean this malware.

What will be done to prevent this in the future?

Considering the most likely breach points involved were provider-centric, my trust in the VM provider is not sufficient anymore to continue using them even for a convenience storage server like we have done. While the CentOS setup is considerably locked down, it will still not prevent local access or control panel access if those are the weak points of their setup. I'll be looking into alternatives that offer storage VMs or other potential solutions that provide an option to archive past-version binaries, symbols, and pre-git source snapshots. If you have any suggestions for such with high level of trust to prevent this kind of nasty situation from happening again, then please send me a private message. Integrity must be high and verifiable.
The archive server will be restored in a more limited capacity using Afterburst over the next couple of days.
"If you want to build a better world for yourself, you have to be willing to build one for everybody." -- Coyote Osborne
Image

Locked