Web server extended outage

Pale Moon releases and site news
(read-only)

Moderator: Indalecio

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 18326
Joined: Sun Aug 28, 2011 5:27 pm
Location: 58.5°N 15.5°E
Contact:

Web server extended outage

Postby Moonchild » Wed Apr 05, 2017 10:54 am

Unfortunately, there has been a rather catastrophic double drive failure in one of the host nodes we use for our web servers at around 9:15 UTC today.

This affects the following services:
  • Pale Moon main website
  • FossaMail main website
  • Pale Moon cross-reference site
  • Pale Moon for Linux main website
  • Pale Moon automatic update site (internal updater)

Expected outage will be 12 hours or more because the server has to be rebuilt. Website data is backed up off-site, and at the moment nothing seems to be lost in that respect.

Message from our service provider explaining what happened:
About an hour ago node Suggan went offline. Upon investigation we have discovered that a second drive has failed in two days. The previous drive, replaced yesterday, had not completed its sync into the RAID arrays, meaning we are left with incomplete data sets. We have managed to rebuild the customer data array and are currently backing this up offsite to verify it's integrity so we can confirm what, if any, data has been lost.

The server itself will need rebuilding, the array the OS was installed on has been entirely lost. This will extend the outage further, updates to this issue can be tracked here: https://client.afterburst.com/network-status/open

[...]

My sincere apologies for this outage, this is the first double drive failure Afterburst has seen.
Light travels faster than sound. That's why some people look brilliant, until you hear them speak.
Image

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 18326
Joined: Sun Aug 28, 2011 5:27 pm
Location: 58.5°N 15.5°E
Contact:

Re: Web server extended outage

Postby Moonchild » Wed Apr 05, 2017 3:20 pm

Some files have been lost - the hosting provider is in the process of copying recoverable files off of the node - we won't know the extent of the damage until that is completed, and we can't take further action until we know the extent of the damage since our next step to recover (either rebuild the web server from scratch or recover from the salvaged set of files) depends on what exactly has been lost.

Sorry for the inconvenience.
Light travels faster than sound. That's why some people look brilliant, until you hear them speak.
Image

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 18326
Joined: Sun Aug 28, 2011 5:27 pm
Location: 58.5°N 15.5°E
Contact:

Re: Web server extended outage

Postby Moonchild » Wed Apr 05, 2017 6:51 pm

Thanks to the excellent service provided by Afterburst, we are almost back up and running completely on a different host node. The Linux site is having database issues that needs to be looked into. The only other site that is still down is our source code cross-reference service, which will have it cross-reference indices regenerated -- this will be done in the course of the coming day as it is a lengthy and processor-intensive task. Data loss occurred; mostly log files and easily-regenerated data, but some things may take some time to sort.
Light travels faster than sound. That's why some people look brilliant, until you hear them speak.
Image

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 18326
Joined: Sun Aug 28, 2011 5:27 pm
Location: 58.5°N 15.5°E
Contact:

Re: Web server extended outage

Postby Moonchild » Wed Apr 05, 2017 11:35 pm

Some bad news -- unfortunately filesystem corruption turned out to be a lot more extensive, and the only way to properly recover will be to rebuild the server.

This will take some time, during which the websites will be unavailable.
Light travels faster than sound. That's why some people look brilliant, until you hear them speak.
Image

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 18326
Joined: Sun Aug 28, 2011 5:27 pm
Location: 58.5°N 15.5°E
Contact:

Update: extended outage

Postby Moonchild » Thu Apr 06, 2017 1:01 pm

An update:

We have a basic page in place for Linux users to get the browser.
Work is still underway to restore the information that has been lost like build instructions. If you need this information right now, your only option is to go to archive.org and grab an archived copy of the content from it.

The cross-reference service for source code is lower priority and will be re-built once the rest is up and running.
Light travels faster than sound. That's why some people look brilliant, until you hear them speak.
Image


Return to “Announcements”

Who is online

Users browsing this forum: AOL Reader [RSS], Blogtrotter [RSS], Digg [RSS] and 43 guests