Downtime expected for various dev services

Discussions about the development and maturation of the platform code (UXP).
Warning: may contain highly-technical topics.

Moderators: trava90, athenian200

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Downtime expected for various dev services

Unread post by Moonchild » 2024-05-13, 11:54

I'll be moving some services to a new DC on newly provisioned servers because we're seriously running out of space/resources for them as time goes on and code bases get more mirrors/forks and become larger in general. Yes, this will be an extra expense for the project but trying to squeeze out more performance and hitching up against a 95% full disk isn't a good idea for the long run, especially on Linux where a full disk can be catastrophic.

I expect some extended downtime to occur for the following: Maybe some other things will be impacted as well but I'll let y'all know if that's the case.

I don't have a set date/time yet for this because provisioning is still in progress but expect this to happen this week and/or the coming weekend.

dbsoft
Project Contributor
Project Contributor
Posts: 455
Joined: 2020-02-21, 17:35

Re: Downtime expected for various dev services

Unread post by dbsoft » 2024-05-13, 21:46

Do I need to start pruning my branches? I've been leaving them around for historical reference purposes... but if they are causing storage problems I can prune them.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-13, 23:02

dbsoft wrote:
2024-05-13, 21:46
Do I need to start pruning my branches? I've been leaving them around for historical reference purposes... but if they are causing storage problems I can prune them.
Nah don't worry about it. having branches in repos doesn't take up so much space. I've already provisioned the new server anyway and it'll have plenty of space (>2 TB) so it's all good.

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1552
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Downtime expected for various dev services

Unread post by athenian200 » 2024-05-14, 02:42

Is there anything I would need to do to facilitate the transition, or would everything mostly just go back to normal after the downtime is over?

If it's just a matter of waiting, I have no problem waiting until everything is back online. Ideally I wouldn't want things to be down for a month or so, but a couple weeks wouldn't be too big a deal, it's not like Epyrus is a fast-moving project that needs 24/7 uptime.

In my mind, standard amount of downtime for scheduled maintenance would be 24 hours or less, and extended downtime would mean anything over 72 hours.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-14, 08:32

I don't think i'll need 24 hours, but expect 48 hours to be safe.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-14, 17:41

I'm doing this one service at a time, tackling (as you may have noticed) the repo service first. it's currently down and I'm transferring all data. I think it should be back up by tomorrow.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-14, 23:13

Repo should be back up. Let me know if there are issues.

User avatar
RealityRipple
Astronaut
Astronaut
Posts: 691
Joined: 2018-05-17, 02:34
Location: Los Berros Canyon, California
Contact:

Re: Downtime expected for various dev services

Unread post by RealityRipple » 2024-05-15, 00:08

Didn't even log me out... which... I guess that's the price of checking the remember me checkbox, but something deep in my programmer's instinct screams out and says sessions should have been stored in volatile space.

dbsoft
Project Contributor
Project Contributor
Posts: 455
Joined: 2020-02-21, 17:35

Re: Downtime expected for various dev services

Unread post by dbsoft » 2024-05-15, 04:19

Still appears offline to me... am I missing something?

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-15, 09:48

RealityRipple wrote:
2024-05-15, 00:08
Didn't even log me out... which... I guess that's the price of checking the remember me checkbox, but something deep in my programmer's instinct screams out and says sessions should have been stored in volatile space.
Not really for the server side of things. Permanent sessions are stored in a permanent location, and if the database is transferred (which it has been) then session information comes along with it. If the client authenticates properly then it doesn't matter what IP the server is on. Also, if you were to store server-side session info in a volatile space only, then how would you ever manage clusters?

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-15, 09:48

dbsoft wrote:
2024-05-15, 04:19
Still appears offline to me... am I missing something?
You may have to flush your DNS cache. IP changed.

EDIT: perhaps IPv6 which you seem to use is slower in updating itself. I verified locally that it resolves to the correct IP (v4 and v6) using my public caching recursive DNS resolver (quad9) so there shouldn't be an issue.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-16, 01:28

Okay, it seems the provider offers IPv6 on their services but it needs to be manually enabled. I was not aware of this and had to dig up the instructions. This is new to me.
I'll work on getting IPv6 working soon.

xref has been transferred and should be working.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-16, 11:57

dbsoft wrote:
2024-05-15, 04:19
Still appears offline to me... am I missing something?
Please do let me know if IPv6 is working - I tested as well as I could (which isn't very much being on an IPv4-exclusive residential connection) and I think it should all be OK and reachable now over IPv6, but a verification would be nice.

User avatar
adoxa
Fanatic
Fanatic
Posts: 198
Joined: 2019-03-16, 13:26

Re: Downtime expected for various dev services

Unread post by adoxa » 2024-05-16, 14:00

Looking at Remote address in Network suggests IPv6 is working fine.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-18, 09:14

add-ons site should be back up, concluding this combined downtime.

User avatar
Basilisk-Dev
Lunatic
Lunatic
Posts: 345
Joined: 2022-03-23, 16:41
Location: Chamber of Secrets
Contact:

Re: Downtime expected for various dev services

Unread post by Basilisk-Dev » 2024-05-19, 00:15

Moonchild wrote:
2024-05-18, 09:14
add-ons site should be back up, concluding this combined downtime.
The Epyrus and Basilisk addons sites are both still down, not sure if you're aware or not.

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-19, 07:08

I wasn't aware since I didn't have an easy way to test before DNS was updated.
epyrus is still not resolving right, but I fixed the bad gateway issue on the basilisk site.

User avatar
Basilisk-Dev
Lunatic
Lunatic
Posts: 345
Joined: 2022-03-23, 16:41
Location: Chamber of Secrets
Contact:

Re: Downtime expected for various dev services

Unread post by Basilisk-Dev » 2024-05-20, 14:20

Moonchild wrote:
2024-05-19, 07:08
I wasn't aware since I didn't have an easy way to test before DNS was updated.
You can test before DNS updates by adding an entry to your hosts file pointing to the IP you provided

User avatar
Moonchild
Pale Moon guru
Pale Moon guru
Posts: 35912
Joined: 2011-08-28, 17:27
Location: Motala, SE
Contact:

Re: Downtime expected for various dev services

Unread post by Moonchild » 2024-05-20, 15:40

I had no reason to believe there was an issue since they are literally pointing to the exact same installation of phoebus, and APMO worked just fine.
I found out what the issue was quick enough though and it's been solved. Epyrus is still down since athenian hasn't updated his DNS yet.

User avatar
athenian200
Contributing developer
Contributing developer
Posts: 1552
Joined: 2018-10-28, 19:56
Location: Georgia

Re: Downtime expected for various dev services

Unread post by athenian200 » 2024-05-24, 21:19

Moonchild wrote:
2024-05-20, 15:40
Epyrus is still down since athenian hasn't updated his DNS yet.
I've updated it now...

Honestly, I'm glad I didn't take on anything more essential than an e-mail client, because I just do not have the attentiveness and discipline to seriously manage a project long-term. Like, if I have a short-term mission that can be over and done with, I'm fine, but I'm really bad at keeping up with something that's entirely up to me...

It didn't help that apparently my domain website has changed their UI and a lot of security stuff since I last logged into it and I had to get them to issue me a new password, etc... it's actually much better now than it was, but it still definitely created some delays.

Post Reply