Tip for PHP sites: disable HEAD

[Moonchild]

Just a tip I want to throw out there for PHP driven sites.

Since there are these days some extremely aggressive web crawlers out there that launch multiple concurrent VM instances to crawl (e.g. DigitalOcean droplets), that rapid-fire requests to websites to crawl them (including fora), it can lead to (effectively distributed) denial of service for regular users. Why? Because these crawlers are designed to index first the URLs that are valid/available by spidering over pages, using HEAD requests.
For PHP (and any other server-side processed scripting), HEAD requests are disastrous because all they do is cause the script processor to do work while the result of the request is never sent or used.

Recently, this very forum came under extremely heavy load because of this bad practice (by a bot called crazywebcrawler) launching so many requests that the server load skyrocketed to 70+ (in case you don't know, that is the number of full-load cores that would be needed to process the tasks without delay). After blocking DigitalOcean as a whole to stop the DDoS I looked into the cause, found this behavior, and instated the following to disable HEAD requests in nginx (inside the .php block):

Code: Select all

    location ~ ^(.+\.php)(.*)$ {
        if ($request_method = HEAD ) {
                # refuse HEAD requests since they cause PHP processing without benefit.
                return 405;
        }
        *** any other fastcgi parameters you need to pass php to the processor ***
    }

If you run a PHP-driven site or other site with server-side scripting, I strongly suggest you do the same for requests to .php (and .asp cgi and similar server-processed scripting extensions) to prevent this kind of abuse/overload.

[John connor]

I've been blocking anything not related to Get or Post since I made my sites. I also used ZBblock, but Zap has been absent from that project so I now use CIDRAM which you can find at Github and talked about here: http://www.spambotsecurity.com/forum/

I block many, many, MANY web hosts in CloudFlare and in CIDRAM including Tor. Many VPNs are blocked, etc. No real reason to hide your IP to me and I provide TLS end to end anyway.

I would also look at Ninja Firewall.

An easy way to deploy this is via htaccess.

Code: Select all

RewriteEngine on
RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK|OPTIONS|HEAD)
RewriteRule .* - [F]

[Moonchild]

Off-topic:
There are many good and legitimate reasons why people wanting to browse the web use a VPN, proxy, or have their traffic otherwise come from a datacenter.
I use a VPN all the time when I'm on the road. Not because I want to hide my IP but because I don't trust the local access point (often wireless hotspots. Even if encrypted it's still going to be operated by someone you have to trust implicitly)
This is a tip to counter "grey area" use and potential DoS from it without locking down your websites -- let's not let this thread end up being a discussion about "the evil net and what you should do to censor its traffic".