Please note: The /member URL has been reconfigured already. The issue is traffic on the non-existent URL.
I have a client who has been running EE 1.6.1. He recently inquired with me about a large number of failed delivery emails he was receiving in response to emails originating from the server. I recommended we go ahead and upgrade his site to get things current and mitigate any issues.
His site was on a basic Media Temple VE server, so the OS & PHP were outdated. As part of his site upgrade I launched things anew on a Rackspace Cloud server running Ubuntu 12.0.4 and the latest apt-get PHP5.
I did not upgrade the original site. I installed EE 2.6.1 and migrated his templates and content in.
He'd been getting "back door" registrations via the default /member
URLs for years. Most were trapped in Pending. Some where in Members. All were generating emails to non-existent inboxes, which was the source of all the email he was getting.
Fast-forward to today:
We're seeing the emails decrease. However, we're seeing a large volume of requests hitting the new site on the now non-existent /member URLs. Allow New Member Registrations
is set to No
and I've changed the Profile Triggering Word
to a non-default value.
During high-volume episodes of requests on these URLs, we're seeing memory usage spike on the server and the server ultimately locking up. I've resized the server from 512MB to 1024MB (after having to reboot it) to mitigate the issue. Most of the time it's functioning quite well within 512MB (and I recently launched another client's site on an identical server with MSM, who's serving up more pages just fine with 512MB). It's the bombardment of requests on /member
URLs that are killing the server, and this wasn't an issue on the old site where those URLs actually functioned.
Has anyone dealt with such an issue? Is there away to mitigate this without continuing to throw more RAM at the server or employ edge caching? I have a hard time viewing this as simply a matter of traffic spiking, because we were seeing this volume of hits on the old server. The only difference was that it was able to respond.
I do have Enable Strict URLs
set to Yes
and the /member
URLs are returning EE's 404 page.
I'm going to look at tweaking settings in Apache next. If anyone can recommend any other solutions from experience, I'd love to hear them. The site went down last night at a time that coincided with a regular spike in emails on the old server. The pattern is clear. The server was at the 1024MB RAM size overnight, so even that was insufficient during peak volume.
I have logrotate enabled for the virtual host's access logs and set a max file size of 5MB to keep things in check there. They look to be hitting 10-20MB over the course of a day, most of which is /member
URLs of various form.
Thanks!
Update: User Agents that I'm seeing
Culprit User Agents are consistently:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)
There are a few /member/email_console/##### requests from:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
But they're not the high-volume ones.
Update 2
Thanks to @jo_nnbc's advice, I have the following now. We'll see if this puts a damper on things:
Redirect 410 /member
BrowserMatchNoCase "Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1; SV1;\)" bad_bot
BrowserMatchNoCase "Mozilla/5\.0 \(compatible; Googlebot/2\.1; \+http\://www\.google\.com/bot\.html\)" bad_bot
Order Deny,Allow
Deny from env=bad_bot
Update 3
After further review, here's what I have in .htaccess:
Redirect 410 /member
BrowserMatchNoCase "Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1; SV1;\)" bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase Ezooms bad_bot
Order Deny,Allow
Deny from env=bad_bot
I've also checked a few of the IPs using whois.sc. They're all from China. I've added the following to my iptables:
-I INPUT -m iprange --src-range 120.40.0.0-120.43.255.255 -j DROP
-I INPUT -m iprange --src-range 59.56.0.0-59.61.255.255 -j DROP
-I INPUT -m iprange --src-range 27.152.0.0-27.159.255.255 -j DROP
-I INPUT -m iprange --src-range 216.244.78.160-216.244.78.175 -j DROP
-I INPUT -m iprange --src-range 110.80.0.0-110.87.255.255 -j DROP
-I INPUT -m iprange --src-range 36.248.160.0-36.248.191.255 -j DROP
-I INPUT -m iprange --src-range 219.154.0.0-219.157.255.255 -j DROP