repo down for the moment

[Moonchild]

The repo server has been taken down for the moment in response to a (by the looks of it) DDoS attack.
I'm investigating which people are trying to ruin someone's Christmas this time around (hint: it doesn't work, I kind of expect attacks around Christmas and it's not like this service is particularly high-availability required anyway).

EDIT: If there are people who do need access, PM me your IP; I can open a gateway for your IP.

[athenian200]

Wow, that's really an awful "Christmas present." I hope you get everything resolved. Thanks for holding down the fort and keeping this project humming along for those of us that need it.

[Moonchild]

I hope you get everything resolved.

Well it's on-going and it does seem to be some form of botnet, with addresses literally from all over the place, randomized user-agents, seemingly with no other reason than to cause load on the server. Yesterday there was a similar issue but that was concentrated almost entirely in China. After geoblocking, they stepped it up a notch employing proxy traffic and more now.
I won't be able to publicly open the repo server until the attack stops, as it's on a low-end box that just can't serve this kind of abusive traffic. I don't have money for more.
As said if you (or anyone else) need repo access, shoot me a private message with your IP and I'll punch a hole in my firewall for you.
I'm monitoring, but can't do much more right now.

[athenian200]

I've been hearing bad rumors that this kind of attack is becoming more prevalent in general, to the point that hosting any kind of independent website is becoming harder and harder.

Is it possible the attacks won't stop, and we would be forced to give up having an independent Gitea server, with our only real option being to move to GitLab infrastructure that's not as vulnerable?

I've always been worried it would get to the point where nation-state level hackers would go after anything they don't like or think they can take down, and everyone would be forced to move "behind the walls" of the biggest tech companies, no longer realistically able to have anything independent because the Internet has become essentially a warzone where you can't just have your own little space anymore.

Anyway, I hope that the attacks are eventually stopped, preferably by the authorities rather than just the hackers getting bored and "letting" us have our server.

[Moonchild]

Is it possible the attacks won't stop, and we would be forced to give up having an independent Gitea server, with our only real option being to move to GitLab infrastructure that's not as vulnerable?

No, there are always options. None of those will have open access to all parts of the repo, though, as that is what is being abused. The Gitea server is just fine, it's just redirecting all traffic that isn't explicitly whitelisted to a different host serving an offline page. nginx has no trouble tossing thousands of 302's out per minute; I just can't have the abuse hit Go and SQL for each request to generate and spit out 100k+ of response data.
I'm not dealing with GitLab, by the way. If anything I will just make access a "by request" thing, authenticated and with clear control over who gains access. Maybe even set up a VPN to do that (you know, a real VPN, not a glorified proxy gateway, i.e. a protected subnet where the repo server runs that you "dial in to")

But for now I'm just sitting this one out. I'm pretty sure they are racking up some costs for this, unlike me, and we'll see who has the longest breath.

EDIT: Thinking about this some more, there's always the option to host a mirror on one of the big sites for people who insist on getting anonymous access for whatever abusive goal they have, but if people want to contribute they can request access to gitea.

[Basilisk-Dev]

It sounds like you have it under control for now, but if there is anything that the community can do to help please feel free to let us know.

[LewisR]

Feel free to ignore/delete my pending post to the Browser development forum about the repo status. Somehow, my searching for repo status missed this thread, and I did not think to check here.

FWIW, my mail server has been getting hit pretty hard the past couple of weeks. I haven't checked other services, though, but I would expect the same. This stuff usually comes in and goes in waves (as we know).

GL

[Moonchild]

Tentatively opening the repo back up. Seems aside from some individually-blocked cloud address ranges, the attack has ceased.
Disappointed in SG fronting for CN (the abuse left is coming from Huawei Cloud ranges in SG datacentres...)

[frostknight]

I'm not dealing with GitLab, by the way.

If you have to pick another repo, besides gitea, might I suggest codeberg?

I have not had many problems with them.

Or if nothing else, just as a mirror.

[Moonchild]

The issue isn't gitea as a software solution. It's fine. My remark was about Gitlab "as a service" not the software.

The issue is that it's being hit with thousands of requests per minute for heavily-processed data, be it deep-linked specific commits and individual files in them, or elaborate issue queries. It's just designed to cause load, nothing else. Similar scripted attacks would run into the same problem no matter what foundry is in use.

Since it's all behind nginx though I can let that do the heavy lifting for filtering of bad requests. Since the traffic started up shortly after I opened up the repo again I'm sure someone is monitoring this and responding. So what I'm doing now is tailoring some filters to counter the worst bad traffic. Unfortunately the nature of the attack can't rely on traditional rate limiting because it's distributed.

[frostknight]

The issue isn't gitea as a software solution. It's fine. My remark was about Gitlab "as a service" not the software.

Hmm, okay well, gitlab is very irritating to use for uxp users in my experience. If you get outdated enough github is less problematic than gitlab for some reason.

The issue is that it's being hit with thousands of requests per minute for heavily-processed data, be it deep-linked specific commits and individual files in them, or elaborate issue queries. It's just designed to cause load, nothing else. Similar scripted attacks would run into the same problem no matter what foundry is in use.

Hadn't known it didn't matter which foundry was in use. But in any case, my point was more that a codeberg is a good platform. But I understand if you prefer self hosting. That is probably the safest way right now.

[Moonchild]

Everyone has their favourites. I looked at codeberg when I wanted to move off of Github, but it really didn't jive with me.

[suzyne]

Off-topic:
Fantastic that this challenge has been resolved. But I am now wishing that I got a screen capture of the moon disintegrating when it was offline, just so I can look at it again. Props to the artist.

[adoxa]

http://offline.palemoon.org/brokenmoon-pal.jpg

[Moonchild]

But I am now wishing that I got a screen capture of the moon disintegrating when it was offline, just so I can look at it again.

https://offline.palemoon.org/

[suzyne]

http://offline.palemoon.org/brokenmoon-pal.jpg

[frostknight]

Everyone has their favourites. I looked at codeberg when I wanted to move off of Github, but it really didn't jive with me.

That makes me wonder what specifically about it you didn't like.

[Moonchild]

I honestly don't remember in detail. It was early days for them at the time, and I didn't an as of then unknown forked version of Gitea run by others as viable. Something just didn't "win" over self-hosting Gitea. maybe I'll give it another look if self-hosting continues to be a regular issue re: attacks.

[Martok]

Hiya,

someone just reached out to me that something apparently didn't come back up:

since the server thing 2 weeks ago, there are no release tarballs attached anymore to tags in repo.palemoon.org

e.g. here
https://repo.palemoon.org/MoonchildProd ... n/releases

and here
https://repo.palemoon.org/MoonchildProd ... P/releases

would you be so kind as to report this issue to Moonchild ?
it prevents users from getting to the source without cloning the entire repo.

Unless that was intentional to reduce load, something might have broken?

[Moonchild]

there are no release tarballs attached anymore to tags in repo.palemoon.org

That was indeed disabled as I dealt with the hammering of the server. I forgot to re-enable it. It's back up again.