(SOLVED) CPU Overload on Modx high traffic content update

Please have a look at the screen shots attached of our cache system settings - maybe something is not set correctly?

Thank you

I'm also experiencing these server overload problems with one of my sites, still not clear what's causing it, but here are some other things to take into consideration:

• replace your getResources call with the pdoResources snippet from the pdoTools extra, it's a lot faster and more efficient
• replace the original phpthumbof snippet with the pThumb + Resizer packages (pThumb works with the same namespace as phpthumbof, so you don't have to change a lot, usually)
• also getCache in combination with other snippets is a very good way to take server load down, just create specific cache partitions for different sections of your site and write a custom plugin to delete these partitions when one of these resources is saved

Cachemaster solves the CPU overhead problem because it doesn't flush the whole website cache on every page save/edit.
The question is:
What do I need to do to keep my home page and some getresources calls up to date without clearing the whole cache? will the updates show eventually without clearing the cache manually? if so then after how much time?

How should I use GetCache to make a separate partition for some resources (home page, newsletter, categories) to clear only their cache every time a new resource is updated/created.
Is there another plugin/snippet that easily does this?

Once I get that done I think my problem might be solved.

Thank you

Quote from: yaronyo at Sep 15, 2013, 08:17 AM

Maybe it also has something to with DirectResize Revo plugin when clearing cache? - Does it clear all the images cache and regenerates them which contributes to the overload?

I haven't used DirectResize, but anything containing the words "plugin" and "image resize" in its description should raise a big red performance flag. Especially with the number of images on your home page.

A plugin like that will run every time the page is accessed, whether it's cached or not, and no matter what caching system you're using. It may not have to generate any images, but it's still got to look through the page's source, find all the img tags and for each one check if the file in the src exists and is the right size, etc. And do it all again the very next time anyone hits the page.

There are a few different versions of DirectResize and I don't know how similar they are to one another, but I took a quick look through the one who's code is on GitHub. Some lines like this are most worrisome:

So if the server's memory limit was set to, say, 256M, this will "increase" it to 64M? Umm, thanks? I don't know about the rest of the program, but as the saying goes "where there's smoke, there's fire" : )

Solved!
1. Cachmaster in order to not flush the whole cache every time there is an update to a resource.
2. Getcache in order to refresh cache of getresources calls only in the important places like home page or sidebar widgets with cache-expires value.

Just tested it out on Peak Traffic and CPU & Memory stays stable around 1 the whole time

Thank You All

This sets the limit of the amount of memory a single PHP script can use, it has nothing to do with the physical memory installed in the server.

Quote from: sottwell at Sep 16, 2013, 02:58 PM

This sets the limit of the amount of memory a single PHP script can use, it has nothing to do with the physical memory installed in the server.

Right. It's obvious what it does; what might not be quite as obvious is what it doesn't do, and why that could cause a rather pernicious problem. Does it first check then value of memory_limit? No, it unwittingly assumes it was less than 64M to start with. Now that's somewhat understandable given that code was written in 2004, but it's a dangerous assumption, especially since it's using GD to resize the images. GD has high memory requirements; it loads the entire bitmap into RAM in order to manipulate an image. To make matters worse, I don't think DirectResize attempts to check available memory vs. image size before resizing it.

Consider this scenario. You've got a nice VPS with lots of RAM. memory_limit is set to 256M or 512M or whatever, plenty of memory. You're using DirectResize. A client uploads a ~3 MB jpeg he got from a photographer. This image is going on the home page. No problem, right? A little big, but DirectResize will size it down!

Well, even though it's just a 3MB file, that's actually a 22 megapixel image. GD will need a memory_limit of at least 128M to handle it. But DirectResize has "increased" memory_limit to 64M. What happens? GD attempts to allocate more memory than PHP is allotted, PHP silently dies and your home page is now a pristine white. If you're lucky there may be an error message in the server log, but chances are there won't be.

Now how much time would this take a web developer to troubleshoot? Lots of dead ends to pursue. Even if you guess it's a memory problem, your reaction would be to increase memory_limit in the ini file, but that won't help since DirectResize is subsequently lowering it. Meanwhile every visitor to your site is getting a white screen and the client is most upset.

@jgrant, this is not pleasant. It is also no surprise. For other Revo discussion related to memory, see:

http://forums.modx.com/thread/?thread=83823&page=1

I'm hoping that the non-specific claim of "improved Manager performance" in Revo 2.2.9 (See http://modx.com/blog/2013/08/29/revolution-2.2.9-sdk-release-and-more/) has the side-effect of lesser peak memory demand.