Site load periodically blowing up, EE Cache related?

Question

I have an error showing up in the logs for a site:

"[Sun Jan 13 13:27:11 2013] [error] [client IP] mv: cannot move /path/public_html/system/expressionengine/cache/page_cache' to/path/public_html/system/expressionengine/cache/page_cache_delete': No such file or directory [Sun Jan 13 13:26:48 2013] [error] [client IP] cannot move /path/public_html/system/expressionengine/cache/page_cache' to/path/public_html/system/expressionengine/cache/page_cache_delete/page_cache'"

'IP' and 'path' represent a real ip and system path.

This site doesn't make use of page caching, but I understand EE can still try to manage it's cache. Normally I wouldn't be concerned, but this particular site has periodic spikes where server load will jump from an average of .5 to 1.5 up to 80 to 100. This crashes the server, and one happened this afternoon.

My host is running loadwatch on the site, and it shows a large number of apache processes without a correspondingly large number of user connections. They also said that there was an overload on the servers memory, which was followed by the load / process spike.

Why do I think these might be related? http://ellislab.com/forums/viewthread/152522/#753282 This forum thread from several years ago on an EE 1 site. It's the only lead I have right now on why this site could be running perfectly fine 99% of the time and then periodically blowing up and taking down a VPS that's more than capable of running it.

Has anyone seen something like this happen on their own sites? If not, any suggestions for how to monitor what's causing spikes like this? My host doesn't seem to have any more to offer as far as tracking, so I can't trace it down to specifically what's causing the spike right now. Any help is appreciated.

Answer 1

Jacob, it occurs to me that CE Cache could come to the rescue here, perhaps in more than one way.

If you are able by intended site content patterns to use it in the sense normally intended, and choose for example the easiest Static Caching mode, you'll have EE actually not serving most of the page hits -- they'll come straight from CE's cache of precompiled html pages, except once each time the cache is broken by a new template or posting edit.

Thus you won't be driving into whatever unforeseen fault mode is causing your immediate problem, and you'll also be getting very high performance. I'm seeing consistent sub-second page loads at any time on a shared server, after going to CE Cache in this simple mode.

Now, what could be causing your situation actually? I would put some money on an old friend, collision of asynchronous processes. If for example one EE PHP instance is still running when another EE instance wants to mess with the cache for itself, that it has a file open (or directory; same thing at internal level in Unix) will prevent other action on that file (or directory). That's pretty likely what you're seeing.

If that's the problem, then the fix would be to assure only one EE PHP can fork at a time on a single server -- and then if you need more horsepower, you'd need a multi-server form of load balancing. The shared database would be the contention issue then, and it has the collision management to handle it, provided transactions are used properly in the EE application.

If you wonder, there are ways to design for servers to handle such contention also, but they would be far too involved for web situations, and I think everyone uses the shared database to handle consistency, which is again what the file system can't do beyond locking you out. I mention this just for background, but presume you'd like to know it, and from a history.

I'd try the CE Cache route, myself -- it's pretty easy to set up, and you can probably work out cache busting settings pretty easily to handle whatever CP activity EE is likely to be providing your client.

Good fortune on this, my friend.

Answer 2

Thoughts about this question have kept burbling up, and while the first answer may be on a good and practical track, I'd like to point to another.

What bothers me particularly seems to be the idea that EE is doing any writing to its cache when simply emitting pages. It really shouldn't, and probably doesn't, until cache intervals wear out. Presumably you've set those relatively long.

However, EE is known to be writing to other tables per site access when under ordinary configuration; in particular to be recording page counts, access IPs and the like. Depending on how transactions may be used, and given the vagaries of ISAM etc. packages, lockups can occur that may occlude multiple access at indeterminate times as you observe.

Thus it may be worth trying some of the overrides Ellis discusses on this page, which turn off hit by hit tracking, unless for some reason you need that.

http://ellislab.com/expressionengine/user-guide/optimization/handling_extreme_traffic.html

These are simple config items, and may clear your problem without having to put a page cache out in front -- though that may be quite useful in its own right if it fits the pattern of your site use, as it takes so much load off the server.

CE Cache can provide more flexibly than simple caches also, if you need to have some part of pages dynamically available, but this would require you have one of the MemCache etc. flavors available on the server, which costs, and that you made some level of effort to arrange a detailed implementation.

Hoping then that the simplest way of these configs can clear the issues for you.