XMPP server (Hookipa) down

[Mæstro]

Although I am fairly sure that this is a server-side problem, I believe it worth reporting here, mainly to ask for advice if the problem persists into tomorrow evening. About 21.35 today by New York time, I was suddenly disconnected from my xmpp.social address. The corresponding website, hookipa.net, is also down, tested in Pale Moon, Waterfox and ping in bash from a variety of VPN servers, and using an online outage checker. I also tried accessing my Jabber account with Conversejs, which reports remote connection failed, and confirming my regular client (Pidgin) works with another Jabber account on another server which I had created for the purpose. The error message is not that in my last Jabber thread, but maemae@xmpp.social/ abgemeldet: Verbindung fehlgeschlagen ([me] logged off: connexion failed).

This is fairly obviously a sudden server outage. Perhaps it is planned maintenance, for it came at three in the morning by the server’s local, Berlin time, but I was not warned about this in advance. The support chatroom is hosted on the downed server, and the regular email to contact for help is on a domain, windfluechter.org, which is also down. As far as I know, Hookipa (as my XMPP host is called) has nowhere else to report on downtime. I welcome correction if I am wrong in my own diagnosis. The only other pertinent fact here is that clicking to reconnect in Pidgin sometimes fails instantly, sometimes only after several seconds; Pale Moon and ping consistently give ‘server not found’ and analogous errors.

While this evidence from several places completes a convincing picture, other than testing when I wake up and again when I return to my computer in the evening if needed, I am unsure how to proceed if it is still down. Creating a new Jabber account on another server is an obvious technical remedy. I was also able to entertain myself tonight without my friends’ company, of course. My problem, if my server is not back online tomorrow, is properly social. There is one friend who understands the problem well, but others who just migrated from Discord this week and might blame XMPP itself or insist on returning to Discord if I explain to them the true nature of the problem so soon, or show up to them again with a new domain. Here, I should note that my friends should all be unaffected by this; none of them use Hookipa addresses. Contacting them by email and describing this accurately, but vaguely, as a ‘sudden internet outage’, is my best idea if the problem has not magically gone away by when it is my regular time to chat with them. (I am lucky that my friends are busy tonight with their own lives.) Have any of you got a better idea to stop this chance event scaring them off?


※As of 1.02 New York time, the Hookipa server is still down. I hope it revives by tomorrow evening, but so far, must plan as if it does not, so the post’s contents stand.

[Moonchild]

It depends on what the problem is. What you describe does sound like there is a critical outage.

Specific error messages matter, though. If it is "server not found" then that means the domain's DNS servers are not working. This could mean their registrar is having trouble with their name servers. It's quite possible the servers themselves are up, but the XMPP clients and web browser can simply not resolve the domain names. I did a quick test and it is indeed the fact that the names for xmpp.social and hookipa.net do not resolve to an IP address (name server requests time out, meaning the name does exist but can't be resolved) I tried both the Pale Moon DNS resolver and cloudflare's.

The resolver for the domains is the same: ns.windfluechter.net - this authoritative name server isn't responding. They don't seem to have a backup/secondary DNS server (ns1.ownisp.de and ns2.ownisp.de seem to be placeholders) so this is their single point of failure. Worse, it seems ns.windfluechter.net itself does not resolve to an IP address, and whois data shows they self-host name resolution for their name server; this is fine but that means they must maintain glue records at their registrar or things can simply never resolve to an IP (if you can't resolve the domain's name server, you can't resolve anything under that domain or any domain it provides DNS for). Their registrar (InterNetX GmbH) is fully operational so this seems to be a problem that windfluechter.net self-hosts everything including DNS and it has failed, and is now in limbo because nobody knows the IP address of their name server.

As to how to translate that to your friends... I don't know. This is a pretty technical problem with windfluechter.net that doesn't easily translate to layman's terms.

[Mæstro]

Thank you for confirming my suspicions and correcting my domain name for Windflüchter! The background information helps for my own reference, for this is not the first time I have experienced sudden downtime with the same error messages like this. Luckily, the server is now live again, sparing me an awkward conversation about this with them tonight. Instead, I can discuss the anime which one wanted me to watch for months, but I only began during this outage.

I will ask through the official support board tonight about how to be notified of server maintenance and the like, in the hope surprises like these should affect me (and by extension, my friends) less. Meanwhile, I imagine that my lay friends would be caught quite off guard if their own XMPP hosts (Yax for one friend, Jix for the others) were to fail for whatever reason. Guarding against that possibility would seem the more important, now that the imminent challenge is gone.

[Moonchild]

windfluechter.net (and .org also belongs to them by the looks of it so you weren't wrong, per se ) should still look into getting their DNS set up correctly with a fallback/secondary DNS server, and make sure they have glue records for their ns.* so that as long as the registrar NS is working (which would be pivotal for any registrar), the name server can always be resolved even if there was downtime.

[Mæstro]

The Hookipa servers suddenly shut down again just now, for about half an hour from 20 New York time. I suspected at first that it was my fault, because it happened at the same time I was changing VPN servers, but third-party checkers could confirm that the server was down again, ie this was coincidence. When this happened today, I suspected that it was Hookipa’s new policy to perform planned maintenance on Mondays at a time when most German users would be asleep, but this theory disagrees with facts I could learn later. In any case, the outage was quite annoying until the server came back online, permitting me to resume my evening plans with friends. Ereyesterday, there had apparently been yet another outage at a time when I was offline, such that I was unaware, but the administrator had this to say then, hidden for its length:

What a day!!!
Ok, we are back and the whole story is quite crazy....
First, the SSDs are exchanged. And then the CephFS has its issues again...
But from the start: First call: $person in front of datacenter, needs to find a way into.
Then the rack with my servers needed to be found. The rack door then needed to be opened remotely.
Next was finding my servers. Then identifying the proper disks. Which caused the issues later, because two wrong disks were pulled.
Usually I read a lot from the disk in question (dd if=/dev/sdb bs=10M for example). This lights up the LED on the disk and should be easy to identify.
Nah! Not so today. Two Ceph disks were pulled.
In order to make it easier for $person I shutdown all the servers, so that the disks can be pulled without further risk.
Worked so far.
But then the new SSDs needed to be mounted on the drive sled. With some screws. The screws were "not good anymore", or in German: verkniesknaddelte Koepfe.
So it took a while until all SSDs were exchanged.
In fact it was planned to last for about 10 mins, it took >1 hour.
And then the servers were powered on again.
Remember that two Ceph disks were accidentally pulled. The led again to the well-known situation that CephFS - where the data lives for Mastodon - was not happy again.
It took again a while to work through the whole mess.
Anyway, the first data disk of Ceph is already setup with SSD as WAL/DB device and speeds things up.
However, until all disks are configured that way, it will take a while. Like a week or so.
The process is pretty straight forward:
take one disk out of service, wipe the disk, add the SSD to the disk and bring it back into service.
Ceph will then copy over the data and when the cluster is happy again, I can proceed with the next disk. With each disk the cluster should be getting a little more snappier and faster.
Hopefully, this was then the last time of Ceph caused outages for a long time!
It took us from December until now to find a date to exchange the SSDs. The old SSDs were used as WAL/DB disks for the Ceph cluster. Without the SSDs there is always a warning about „slow ops“. When there are too many slow ops, CephFS refuses to work, as it seems. Especially after reboots for example because of Linux kernel security updates the number of slow ops increases. Last week there were about 1100 slow ops. Usually only single digit numbers.
Luckily I could buy the new SSDs before the prices skyrocketed later.

At the time of writing, I do not know whether last week’s or today’s problems were related to ereyesterday’s incident, but I can say from the support chatroom’s logs, or at least those I can read, that I am not the only user strongly considering switching to another Jabber server, preferably one which I can contact by email (Hookipa gives none on its site!) and where nightly maintenance falls when I am offline anyway. Japan might suit; my available window in the States (19?–25) corresponds to Japanese waking hours. For me, the problem in switching accounts is half technical, half social.

On the technical side, there is the matter of migrating my contacts, and most importantly ensuring that my chat logs in Pidgin stay properly combined, or else can be merged. That is, any future conversations with Alice from my future Jabber account on another server should appear continuous with my extant logs under my old one under Alice’s chat window, instead of having two separate records for Alice–Mæ/Hookipa and Alice–Mæ/Step.

On the social side, I would need to explain to that one, iPhone-bound friend who switched a fortnight ago, and who only uses Jabber to talk with me, why I have switched accounts in a way that does not erode what little confidence he places in XMPP, or rather, Monal. From his perspective, Monal is just another, ‘inconvenient’ app; I never explained to him even the basics of XMPP’s nature, to simplify the transition for him. The description in the App Store is accurate when it claims you can use Monal without caring what Jabber even is!

The timing is especially awkward, for as of today, returning to Discord is now literally the stuff of my nightmares. I dreamt last night that, for obscure reasons, I needed to log back into my old Discord account which is, in fact, pending deletion and due to be deleted tomorrow. In the dream, that friend and others could see me ‘returning to Discord’ on that business and pressured me successfully into returning to that loathsome site. My sincere belief is that this dream is meant to warn me to tread with caution about such matters.

[Moonchild]

I don't think you necessarily need to explain in detail what is going on. Just let your contacts know there were some issues with the current address and you need to change to a new one. If someone is really interested in knowing the details they can ask.

[Mæstro]

Fair enough. I tend to overthink these things. Happily, I have a clearer picture of what happened, thanks to the sysadmin’s kind explanations in the support room. He was migrating gradually six hard drives in all, hence the irregular outages over time, and the last of these should have been completed earlier today. I was pointed to the official status checker. Since I should not need a new Jabber account at this time, I should not worry about this. My friends should have more time to become accustomed to Jabber, and my old Discord accounts are now gone for good, the grace period to reverse deletion having expired. Any incident like this in future should be easier to manage.