Web server extended outage

Pale Moon releases and site news
(read-only)
User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Web server extended outage

Unread post by Moonchild » 2017-04-05, 10:54

Unfortunately, there has been a rather catastrophic double drive failure in one of the host nodes we use for our web servers at around 9:15 UTC today.

This affects the following services:
  • Pale Moon main website
  • FossaMail main website
  • Pale Moon cross-reference site
  • Pale Moon for Linux main website
  • Pale Moon automatic update site (internal updater)
Expected outage will be 12 hours or more because the server has to be rebuilt. Website data is backed up off-site, and at the moment nothing seems to be lost in that respect.

Message from our service provider explaining what happened:
About an hour ago node Suggan went offline. Upon investigation we have discovered that a second drive has failed in two days. The previous drive, replaced yesterday, had not completed its sync into the RAID arrays, meaning we are left with incomplete data sets. We have managed to rebuild the customer data array and are currently backing this up offsite to verify it's integrity so we can confirm what, if any, data has been lost.

The server itself will need rebuilding, the array the OS was installed on has been entirely lost. This will extend the outage further, updates to this issue can be tracked here: https://client.afterburst.com/network-status/open

[...]

My sincere apologies for this outage, this is the first double drive failure Afterburst has seen.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Web server extended outage

Unread post by Moonchild » 2017-04-05, 15:20

Some files have been lost - the hosting provider is in the process of copying recoverable files off of the node - we won't know the extent of the damage until that is completed, and we can't take further action until we know the extent of the damage since our next step to recover (either rebuild the web server from scratch or recover from the salvaged set of files) depends on what exactly has been lost.

Sorry for the inconvenience.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Web server extended outage

Unread post by Moonchild » 2017-04-05, 18:51

Thanks to the excellent service provided by Afterburst, we are almost back up and running completely on a different host node. The Linux site is having database issues that needs to be looked into. The only other site that is still down is our source code cross-reference service, which will have it cross-reference indices regenerated -- this will be done in the course of the coming day as it is a lengthy and processor-intensive task. Data loss occurred; mostly log files and easily-regenerated data, but some things may take some time to sort.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Web server extended outage

Unread post by Moonchild » 2017-04-05, 23:35

Some bad news -- unfortunately filesystem corruption turned out to be a lot more extensive, and the only way to properly recover will be to rebuild the server.

This will take some time, during which the websites will be unavailable.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35474
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Update: extended outage

Unread post by Moonchild » 2017-04-06, 13:01

An update:

We have a basic page in place for Linux users to get the browser.
Work is still underway to restore the information that has been lost like build instructions. If you need this information right now, your only option is to go to archive.org and grab an archived copy of the content from it.

The cross-reference service for source code is lower priority and will be re-built once the rest is up and running.
"Sometimes, the best way to get what you want is to be a good person." -- Louis Rossmann
"Seek wisdom, not knowledge. Knowledge is of the past; wisdom is of the future." -- Native American proverb
"Linux makes everything difficult." -- Lyceus Anubite

Locked