site slowdown after clearing cache

Hi Shaun,

here’s a log of all the 97930 queries that run every time I save a document. In this case, I’m trying to save the document with the id 39723 (which occurs in just a few of the queries).

The same thing happens to my co-workers, so I don’t think it has to do with my Apache setup. We are running MODx Revolution 2.0.3-pl on Mac OS X and XAMPP.

Thanks.

You should definitely try the latest trunk version (to be released as 2.0.5) to see if the issues persist. 2.0.4-pl2 had some significant improvements over 2.0.3 and 2.0.5 repeats this again.

Hi Ryan,

thanks for your reply. How can I try the latest trunk version? Do I have to check it out from Subversion?

Thanks.

yeap, this is pretty much a showstopper for large sites.

The Evolution was becoming pretty slow with around 10K of docs (and i know it wasnt designed to handle more than 5K), but Revolution was advertised as a system without a hard document limit. Hopefully this will be fixed in the next updates.

I think I have to install it from git to see the latest code changes.

Would I simply do:



as described here:

http://rtfm.modx.com/display/revolution20/Git+Installation

It’s confusing, because there is also a section on installing from Subversion:
http://rtfm.modx.com/display/revolution20/Using+MODx+Revolution+from+SVN

I am more familiar with Subversion, so I’d prefer to install it from there.

Are you using a Subversion-to-git bridge?

Is the following Subversion branch the same as the "master" git repository?

http://svn.modxcms.com/svn/tattoo/tattoo/branches/2.0/

Thank you.

No, The Subversion repository is no longer up to date and closed to new commits. Everything past 2.0.0-pl release is available only on GitHub.

That said, things going into next release are in the develop branch, not master which is the default branch representing stable production releases. If you don’t want to mess with Git or clone the repository, you can get a download of the current develop branch by going to https://github.com/modxcms/revolution/tree/develop and click on the Downloads button.

Then just follow the build instructions starting at http://rtfm.modx.com/display/revolution20/Git+Installation#GitInstallation-RuntheBuild -- just note this is not a complete build and does not include compressed/concat js/css files.

I’ll update the documentation this weekend with more information, including how to build a distribution package (including compressed/concat js/css files) with Phing. The actual build scripts used to use Ant, but these were just recently converted to Phing in the develop branch. If you have Phing (or want to install it from PEAR), you can do the complete build of MODx Revolution from source code easily.

FWIW, we’ll also be announcing at least weekly distribution packages from develop starting next week.

Can the size of the cache file be an issue in this case?

i just checked the size of: /core/cache/web/context.cache.php and its ~14Mb - seems like when you clear the cache file, this file needs to be recreated - is that the case?

Yes, that is the problem, and the problem is simple: you have too many Resources in a single Context. To be honest, if you have more than a 100 of similar pieces of data to render views for, then you should consider storing that data and rendering the dynamic views through a single Resource. You can try disabling the cache_context system setting, which prevents the caching of the Resource map for the context (you will not have the 14Mb file at all) and see what kind of performance you might get. It will definitely increase the per-page load time for anything that needs to generate a link to a Resource, and you’ll need to make sure you Resource caches are optimized so the links are not being generated every request.

That said, there are some improvements coming in 2.1 that will speed up link generation and this will improve the speed of re-generating the context cache map of Resources as well as dynamic on-demand generation when cache_context is disabled.

Wait, sorry, am i reading this correctly? you have too many Resources in a single Context
So basically what you saying is that we are too big for MODx and the system DOES have a document limit after all...

We will definitely try to disable that cache setting and see what happens, although i dont see this helping on a 20mil+ pviews/month site.

I really hope the context caching will be addresses in the next releases (maybe dont store all context info in one file and split it into individual cache files?).

Thanks!

Quote from: bakalek at Nov 11, 2010, 12:34 PM

Wait, sorry, am i reading this correctly? you have too many Resources in a single Context
So basically what you saying is that we are too big for MODx and the system DOES have a document limit after all...

There is always a limit of some kind. And scalability is never linear. You seemed to ignore the most important part of my post though, which was that representing data as MODx Resources is the problem. MODx Resources are intended for presentation, or creating views into data. The approach of representing data items as MODx Resources is convenient in cases where quantities are limited, but it will never scale.

Quote from: bakalek at Nov 11, 2010, 12:34 PM

We will definitely try to disable that cache setting and see what happens, although i dont see this helping on a 20mil+ pviews/month site.

I really hope the context caching will be addresses in the next releases (maybe dont store all context info in one file and split it into individual cache files?).

Splitting the cache files is why contexts were created. Beyond this, the only thing that can improve the performance is speeding up the process of building the Resource links on-demand, which is what I referred to when I said improvements were coming. With that kind of traffic, you likely need to consider some more robust caching strategies.