Optimize! Experimenting with full-caching

I've been getting very poor performance from my pages so I'm trying to figure out where the bottlenecks are. My goal is to understand better what it takes to improve performance and then balance that with the need for dynamic data.

I've set values for all system settings for cache. Cache is on with a 30 second refresh just so I can test quickly.

I've used the CacheClear snippet, hitting it a couple times to remove existing cache.

My pages initially load in about 18 seconds. Given the modest nature of these pages, that's awful. On refresh I get 3.5-5 second turn around of the base page. With anywhere from 5-20 images, each loading in 100-300ms, that's adding a few seconds on top of that. I'll do what I can to optimize the images.

But the pages still seem to be cached on the server. In the return headers I get:
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

I did a search for "[[!", and then went around to edit out the exclamation marks.

I don't expect too much delay from my hosted server. The rest of the internet is quite zippy, but my ModX site is not, and my Manager is a real slug.

I'm already being selective about the scripts I'm loading. At some point soon I'll look at MinifyX too. Scripts/CSS don't seem to be hurting the page much.

What else should I be looking at on the MODx side?

Thanks!

I'll add that I've turned on compress_js and compress_js_groups, but the js files are not being compressed for the site pages.

After doing that, the manager no longer loads properly in any browser - and at best it's taking about. I've been suspecting AjaxManager is causing some grief. This environment wasn't misbehaving before I loaded that and a couple other addons today. I'm at the point now where my dashboard loads but I can't see the tabs for resources or anything else. So the environment is stuffed but I'll work that out separately.

So is compress_js expected to work?

Thanks.

As far as I am aware, the compress_js is only for Manager pages.

Well, I went into the database and manually disabled compress_js and AjaxManager. I was able to get my Manager back. There are other issues but for this thread, compress_js is now off and I'm looking forward to figuring out what else can be done.
Thank you.

MODX is fairly demanding when it comes to server resources, so if you're on an overcrowded shared host, things will be pretty slow.

MODX page loads in the front end have a lot to do with what you're calling. PHX-style conditional output modifiers can be very slow and so can pages with many images (especially thumbnails) and complex getResources calls.

You might check out MODX Cloud, which is *very* fast compared to most servers (especially in the MODX Manager).
https://modxcloud.com/

I just saw this discussion:
http://forums.modx.com/thread/81754/cache-clarification-about-revolution

Dove-tailing with that here, is there any trace feature that can tell us exactly which elements are being re-processed and which are being cached, along with a down-to-the-millisecond timing for each element? For example:

2013/01/21 12:39:33.0233 : snpAaaa (C) << snippet cached, skipped
2013/01/21 12:39:33.1511 : snpBbbb (C) << also cached
2013/01/21 12:39:33.7236 : chkCccc (U) << chunk not cached, needs to be processed
2013/01/21 12:39:36.8198 : chkDddd (C) << last element took 3.1 seconds

With that kind of data it would be so easy to pinpoint bottlenecks on the server side, which is impossible on the client. I don't know anything about the internals of the life cycle processor but I have to believe there is a central controller which could log all calls for all or specified sessions or requests. And I understand that could kill performance on its own but it would only be turned on for testing like this.

As you might be able to tell from above, I'm confused about how the cache mechanism works where some page elements are cached and some are not. Does it really cache down to the element level to pull in specific cached page segments? Or does it consider the whole page uncached if any element is found to be uncached? I'm thinking the HTTP response header must show all pages as no-cache unless they're really static resources. Any [[!element]] in the stream could completely invalidate the cache for the entire page.

I've read a good amount on cache for MODx but obviously I'm still not "getting" how this deep and often-discussed mechanism works. Is there a single, comprehensive document on this topic? (I'd rather know where to fish than to be tossed a fish.)

Thanks!!!

You can look at the files in the core/cache directory to see what's being cached and how. The numbers in the file names correspond to the IDs of the Resources and Elements.

As I understand it, in a cached page, template, or chunk with uncached tags, the page is stored in the cache with the tags unprocessed. They're processed each time the page is loaded.


In my experience, the cache is pretty fast. Simplifying your tags and replacing complex tags with custom snippets has a bigger effect than tuning the caching (though both are important).

Images are a particular issue since the creation of thumbnails and resizing are often done on-the-fly each time a page is requested. If you comment out your image tags in the HTML you can see how much time they are eating.

I think there are also some pretty good timing add-ons for FireFox that might be informative.

OTOH, if your host is slow, nothing you do will help that much. [ed. note: BobRay last edited this post 11 years, 3 months ago.]

Quote from: captainstarbuck at Jan 21, 2013, 08:40 PM

...is there any trace feature that can tell us exactly which elements are being re-processed and which are being cached, along with a down-to-the-millisecond timing for each element? For example:

2013/01/21 12:39:33.0233 : snpAaaa (C) << snippet cached, skipped
2013/01/21 12:39:33.1511 : snpBbbb (C) << also cached
2013/01/21 12:39:33.7236 : chkCccc (U) << chunk not cached, needs to be processed
2013/01/21 12:39:36.8198 : chkDddd (C) << last element took 3.1 seconds

With that kind of data it would be so easy to pinpoint bottlenecks on the server side, which is impossible on the client. I don't know anything about the internals of the life cycle processor but I have to believe there is a central controller which could log all calls for all or specified sessions or requests. And I understand that could kill performance on its own but it would only be turned on for testing like this.

There is some of that information available at various logging levels, though maybe not exactly what you are looking for here. That kind of logging information would certainly be helpful, though you can get that at the PHP level using xDebug's profiler, if you have access to a development environment with xDebug installed.

In the meantime, you can use something like this to time the execution of non-cacheable Elements:

http://www.jasoncoward.com/technology/2010/10/the-executioner-snippet.html

Quote from: captainstarbuck at Jan 21, 2013, 08:40 PM

As you might be able to tell from above, I'm confused about how the cache mechanism works where some page elements are cached and some are not. Does it really cache down to the element level to pull in specific cached page segments? Or does it consider the whole page uncached if any element is found to be uncached? I'm thinking the HTTP response header must show all pages as no-cache unless they're really static resources. Any [[!element]] in the stream could completely invalidate the cache for the entire page.

I've read a good amount on cache for MODx but obviously I'm still not "getting" how this deep and often-discussed mechanism works. Is there a single, comprehensive document on this topic? (I'd rather know where to fish than to be tossed a fish.)

There is not a single, comprehensive document, though there are some good posts regarding it (listed below).

FWIW, all PHP pages, especially virtual pages in a Content Management System, must return an HTTP no-cache response, or the script will not be executed again until the client expires it's local cache, regardless of what happens on the server.

MODX itself caches the Resource data, the cacheable output from the Resource (already processed), the output from all unique Tag strings that are cacheable, as well as the Element data for all Elements used on the Resource either via Tags or via Plugin Events (including non-cacheable Tags). So instead of grabbing all the data objects needed to construct the cacheable output, requests to cached Resources grab this partial-page cache and proceed to process all the non-cacheable Elements. If none of those non-cacheable Elements require a query to the database internally (remember, their source is cached into the Resource cache, so no database hit is required on a cached page load by default to load the Elements used on it), and you do not require sessions or are you not using the default database-driven session handler in MODX, you can avoid a connection to the database completely.

I hope this helps a little...

https://www.markhamstra.com/modx-blog/category/Caching/
http://modx.com/blog/2012/10/29/optimization-with-getcache-and-custom-cache-partitions/

Good stuff. Going to take a while to absorb this - and then I'll be back for more. LOL

Jason, thanks for that blog. I also find myself linking to my blogs of years gone by for colleagues who are still asking questions that never seem to go away.

Thanks all!