A site in Algeria was sending thousands of requests to a server that I manage. On various pages of their site they had hotlinked an image from our site. However, our site did not have that image on it, so I have no idea what it trying to do. Since it was getting a 404, it was not really bogging down the server, but I would rather it get stopped at the firewall level. That way it would not slow down Apache at all. To do that I used Fail2Ban. Here are the steps I used to do this.
Since I was getting requests from hundreds of IPs in North Africa, I could not really block a few and be fine. So I used fail2ban to read the logs and block any IP that was requesting the file for 20 minutes. Since 99% of our traffic is from North America, I figured there was a 0.00000001% chance of someone hitting the bad referer site, then within 20 minute trying to hit our site for real. So here is what added to the bottom of my /etc/fail2ban/jail.local file:
[apache-url] enabled = true port = http,https filter = apache-url logpath = /var/log/apache*/*access.log maxretry = 1 bantime = 1200 action = iptables-multiport[name=url, port="http,https"]
Then I created /etc/fail2ban/filter.d/apache-url.conf with the following contents
[Definition] failregex = ^www.yourdomain.com:80 .*the_invalid_file.*$ ignoreregex =
That is it, now anyone who hits that bad url, will be blocked for 20 minutes. This took down the hits on my server substantially.