4

Please note: The /member URL has been reconfigured already. The issue is traffic on the non-existent URL.

I have a client who has been running EE 1.6.1. He recently inquired with me about a large number of failed delivery emails he was receiving in response to emails originating from the server. I recommended we go ahead and upgrade his site to get things current and mitigate any issues.

His site was on a basic Media Temple VE server, so the OS & PHP were outdated. As part of his site upgrade I launched things anew on a Rackspace Cloud server running Ubuntu 12.0.4 and the latest apt-get PHP5.

I did not upgrade the original site. I installed EE 2.6.1 and migrated his templates and content in.

He'd been getting "back door" registrations via the default /member URLs for years. Most were trapped in Pending. Some where in Members. All were generating emails to non-existent inboxes, which was the source of all the email he was getting.

Fast-forward to today:

We're seeing the emails decrease. However, we're seeing a large volume of requests hitting the new site on the now non-existent /member URLs. Allow New Member Registrations is set to No and I've changed the Profile Triggering Word to a non-default value.

During high-volume episodes of requests on these URLs, we're seeing memory usage spike on the server and the server ultimately locking up. I've resized the server from 512MB to 1024MB (after having to reboot it) to mitigate the issue. Most of the time it's functioning quite well within 512MB (and I recently launched another client's site on an identical server with MSM, who's serving up more pages just fine with 512MB). It's the bombardment of requests on /member URLs that are killing the server, and this wasn't an issue on the old site where those URLs actually functioned.

Has anyone dealt with such an issue? Is there away to mitigate this without continuing to throw more RAM at the server or employ edge caching? I have a hard time viewing this as simply a matter of traffic spiking, because we were seeing this volume of hits on the old server. The only difference was that it was able to respond.

I do have Enable Strict URLs set to Yes and the /member URLs are returning EE's 404 page.

I'm going to look at tweaking settings in Apache next. If anyone can recommend any other solutions from experience, I'd love to hear them. The site went down last night at a time that coincided with a regular spike in emails on the old server. The pattern is clear. The server was at the 1024MB RAM size overnight, so even that was insufficient during peak volume.

I have logrotate enabled for the virtual host's access logs and set a max file size of 5MB to keep things in check there. They look to be hitting 10-20MB over the course of a day, most of which is /member URLs of various form.

Thanks!

Update: User Agents that I'm seeing

Culprit User Agents are consistently:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;)

There are a few /member/email_console/##### requests from:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

But they're not the high-volume ones.

Update 2

Thanks to @jo_nnbc's advice, I have the following now. We'll see if this puts a damper on things:

Redirect 410 /member

BrowserMatchNoCase "Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1; SV1;\)" bad_bot
BrowserMatchNoCase "Mozilla/5\.0 \(compatible; Googlebot/2\.1; \+http\://www\.google\.com/bot\.html\)" bad_bot
Order Deny,Allow
Deny from env=bad_bot

Update 3

After further review, here's what I have in .htaccess:

Redirect 410 /member

BrowserMatchNoCase "Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1; SV1;\)" bad_bot
BrowserMatchNoCase MJ12bot bad_bot
BrowserMatchNoCase Ezooms bad_bot
Order Deny,Allow
Deny from env=bad_bot

I've also checked a few of the IPs using whois.sc. They're all from China. I've added the following to my iptables:

-I INPUT -m iprange --src-range 120.40.0.0-120.43.255.255 -j DROP
-I INPUT -m iprange --src-range 59.56.0.0-59.61.255.255 -j DROP
-I INPUT -m iprange --src-range 27.152.0.0-27.159.255.255 -j DROP
-I INPUT -m iprange --src-range 216.244.78.160-216.244.78.175 -j DROP
-I INPUT -m iprange --src-range 110.80.0.0-110.87.255.255 -j DROP
-I INPUT -m iprange --src-range 36.248.160.0-36.248.191.255 -j DROP
-I INPUT -m iprange --src-range 219.154.0.0-219.157.255.255 -j DROP
Improve this question

4 Answers 4

Reset to default
4

Are you able to see where they are coming from? Is it coming from random user agents or specific ones? Or is the user agent blank?

htaccess blacklist any strange user agents if that helps. Or block by some country ip addresses for now if it is specific.

Improve this answer
4
  • I posted the user agents I'm seeing in the original post. Yes--There is a clear culprit. I'm going to block that to see if I can curb the requests. Thanks!
    – Mark J. Reeves
    Jun 6, 2013 at 18:48
  • They still hit the server. Might help to notify the host too to see if there is anything they can do to block it. Also, perhaps can set up htaccess to just plain block access to old /member url (that was from the old site) and send to error code 410 "gone".
    – jo_nnbc
    Jun 6, 2013 at 19:06
  • It's a Rackspace Cloud server - It's on me to set it up. I just blocked the requests to those user agents and tested by setting the same UA in Safari. If I could narrow down IPs or maybe even with UA, and could spend more time with this, I could try doing it in iptables.
    – Mark J. Reeves
    Jun 6, 2013 at 19:14
  • 1
    I've had to deal with this recently too, and I just ended up using .htaccess to block the entire subnet that the requests were coming from. Maybe a touch aggressive, but it sure helped with server load! So if you can isolate it to a specific IP range, that's an option.
    – Mark Drzycimski
    Jun 6, 2013 at 19:34
2

I would suggest installing Fail2Ban - it is a process that will monitor your logs and ban (via iptables) IPs that are trying to log in at excessive rates, or otherwise exploit your server.

Linode has a good writeup on how to set it up: https://library.linode.com/security/fail2ban

Improve this answer
1
  • Thanks! I checked a few IPs and found them in ranges in Chinese ISPs. I put those ranges into iptables myself. I've mitigated the issue quite a bit. I think enough effort has been spent at this point to get it under control and help it die out.
    – Mark J. Reeves
    Jun 7, 2013 at 13:12
1

Add this to your config.php:

$config['profile_trigger'] = md5(mt_rand());

This will hash the /member value and is completely unguessable by bots.

Improve this answer
5
  • 1
    Thanks for the quick reply. I noted that I've already changed the /member value via the CP and that the backdoor is closed. The issue is the volume of requests to that URL. It's traffic that's affecting the server.
    – Mark J. Reeves
    Jun 6, 2013 at 18:34
  • Ah, crap. You’re absolutely right. I was just all too excited to share the config value. Are you using anything like Bad Behavior or anything to block IP addresses from known bots?
    – Christopher Kennedy
    Jun 6, 2013 at 18:37
  • No, that's the sort of advice I need, though. Unsure how that will mitigate the memory spikes, though. I'll take a look - Thanks!
    – Mark J. Reeves
    Jun 6, 2013 at 18:39
  • Looking at Bad Behavior, that's post-Apache. These requests are zeroing right in on the /member URLs that were available on the old site. I think the problem is in Apache fielding the requests.
    – Mark J. Reeves
    Jun 6, 2013 at 18:40
  • Wow yeah, that’s a terrible spot to be in. Please post if you find any sort of automated method to curtail this behavior at that level. It would definitely come in handy for others in the future!
    – Christopher Kennedy
    Jun 6, 2013 at 18:42
0

EE's 404 page is still an EE/PHP page generation. You might lessen the server load even more with a command like this in your .htaccess file:

redirect 301 /member http://pickabaddomain.com

That way Apache is telling them to bug off.

Improve this answer
2
  • We're using the 410 "gone" response for that. The other curious thing is that the old site wasn't buckling under this traffic like the new one. It was returning EE pages for /member URLs.
    – Mark J. Reeves
    Jun 6, 2013 at 19:49
  • 410 is probably a better choice. For the old site vs. new site, it could have to do with the addons installed, or what kind of caching is enabled. Most managed hosting solutions optimize the page loads. It's a bad example, but GoDaddy and HostGator consistently use less memory for the same page load (cloned database) than my dedicated server. I haven't attacked that yet.
    – EpicVoyage
    Jun 6, 2013 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.