Performance of MODx for larger sites

Optimization and component flexibility are almost always at odds in the world of web application development, just as are the finer qualities of optimization itself, i.e. memory footprint vs. speed of execution. I have done a lot of experimenting over the past 6 months focusing on these issues in regard to the current MODx code. The ability to create a menu quickly can be addressed by removing all need for SQL, and making all the metadata of each MODx document available in a preloaded array, similar to how it works now. But as already pointed out, with large sites, this does not scale well. But neither does executing queries to search for individual resources in the database when building internal links, or menus, etc. Further, knowledge of and access to the site structure is a very common and important requirement for a lot of add-on components, even beyond the menu builder.

My solution 6 months ago, when developing the initial rewrite of MODx using OO principles, was to compromise by allowing a site to be divided into contexts. These contexts could preload a map of their section of the site, similar to how siteCache does now, but including all necessary metadata for building common menus and links within that section. In this way, if you have 20,000 documents, you could organize those into logical sections and define a context for each, so that the memory load/speed of execution balance is maintained by only preloading the document metadata array for that context (perhaps plus any defined without context, or within a global context).

Let me know your thoughts on this as I work on completing the 1.0 rewrite; if you do have alternative suggestions or ideas in this regard, let’s discuss them now.

And yes, we can definitely optimize the SQL in DropMenu to do all of the selection, but depending on table indexing and other SQL factors, it may be faster to just load the data less exclusively and loop through like it does. And caching does address it very well if you don’t mind refreshing your entire site cache whenever your site structure changes; this is truly the best way to deal with it.

Hi all,
sorry I am short in time at the moment.

Perhaps I will find some time for further testing on weekend
else I will do it next week.


One quick note:

we can definitely optimize the SQL in DropMenu to do all of the selection, but depending on table indexing and other SQL factors, it may be faster to just load the data less exclusively and loop through like it does

In my special case with 20000 pages loaded and without menu caching it was an mysql speed improvement of more than one second when excluding hidden elements by query!

Thanks a lot for all this input.

Regards,
Tom

Friend, did we make some particular database optimizations? Have we have a closer look on how indexes are set up or is it just like how it was x years ago? The indexes should convert all waiting time to be logarithmically dependent on th number of db entries, so I would not expect mysql time increasing so quickly...

The other part of this discussion is how large numbers of documents affects the performance of the manager...

So it sounds as if modifying the DropMenu snippet to use a specific mySQL query that eliminates hidden docs instead of teh GetActiveChildren API makes a huge difference when creating those menus. Seems like there’s no reason not to do that then. Perhaps there should be more parameters that can be passed to that function to allow for more specific queries?

I hear that the new WayFarer menu snippet will be awesome...