MODx limits on the number of documents

Quote from: bakalek at Feb 21, 2008, 05:32 PM

Do unpublished pages count toward the ’maximum’ page limit?

They do not increase the cache file size, but they may have other performance affects since they are additional rows in the database.

Quote from: bakalek at Feb 21, 2008, 05:32 PM

Are there any tricks to reduce the cache size?

Sure; every chunk, snippet, and plugin in your site is included in your cache file. One trick I often use to reduce the impact of snippets is to use a single snippet to include all snippets as php includes. This means only one, tiny snippet in my cache file. You could avoid the same with chunks by making your snippets use php includes instead of chunks for templating.

In the 0.9.7 series, this will not be as much of an issue as the cache has been organized into logical sections and a number of new caching options will allow you to tune the cache to best fit the size of your site and the environment it is deployed in.

Quote from: bakalek at Feb 21, 2008, 05:32 PM

if we dont use the internal search - does it help with performance?

Internal search? Can you be more specific? Search does not really have an impact here, other than the size of the full-text index on the site_content table, which of course will grow with the size of the table. If you don’t use any of the search snippets that utilize this search index, and don’t need to use the manager search mechanism, you might be able to remove this full-text index, but I’ve never tried it.

Thank you very much!
i remember looking at the ’snippet/chunk includes’ at some point - got to find the right threads now...

As far as the search goes: we went with Google search since we have modx and wordpress running together and we needed a solution to pull results from both aps. So now we deactivated ’searchable’ by default, hoping it will help with performance a bit.

Quote from: bakalek at Feb 21, 2008, 06:32 PM

Thank you very much!
i remember looking at the ’snippet/chunk includes’ at some point - got to find the right threads now...

As far as the search goes: we went with Google search since we have modx and wordpress running together and we needed a solution to pull results from both aps. So now we deactivated ’searchable’ by default, hoping it will help with performance a bit.

You might also try dropping the fulltext index on site_content table as Jason mentions above (if you haven’t already done so) -- content_ft_idx is the name of the index. I notice that on a relatively small site of mine, that index is over 60K so on a large site it might that doesn’t need searching, removing it might help performance. You can always put if back if it turns out that you need it.

Bob

I’ve found that the biggest bottleneck is the site cache, which needs to be read and parsed whenever a page is sent. In practice this can sometimes take a lot of CPU power at the very beginning of the process, but if you’re on a server with enough CPU to handle it then the time factor is often not that significant. For a few milliseconds CPU use spikes up while it’s parsing the cache file, but then it drops down again and things speed along.

So if you have an underpowered server or receive a whole lot of simultaneous traffic, then this can cause resource problems if the CPU is overwhelmed. Jason already mentioned some ways to reduce the size of the site cache. I usually just move my biggest snippets and plugins to include files, but the smaller you can make your site cache file, the less of a bottleneck it will be.

I’ve had some sites with 3,000 to 5,000 docs and heavy (30GB/month) traffic, and MODx has never slowed down noticeably on any of them. We had one site that had an automatic hair-trigger DDOS defense system that was a major poin in the butt once, but that was more a problem with this particular host than anything else. When the CPU would spike over 1% they would shut down our site, which turned out to be an untenable situation.

I insert the following code into all of my templates to get an idea of parse time and source:
<!-- Queries: [^q^] | Query Time: [^qt^] | Parse Time: [^p^] | Total Time: [^t^] | Source: [^s^] -->

Personally I’ve found that MySQL queries are generally so fast that I don’t bother to do anything other than optimize the tables now and then, whereas PHP parsing can take a lot longer, but your mileage may vary.

Yeah, my cache file size is 900kb - i think its a bit too much. i’m optimizing all the snippet/chunk calls now.

This is what i got on the home page (cached and non cached).

<!-- Queries: 8 | Query Time: 0.0058 s | Parse Time: 0.1764 s | Total Time: 0.1822 s | Source: cache -->

<!-- Queries: 3873 | Query Time: 2.3672 s | Parse Time: 8.9025 s | Total Time: 11.2697 s | Source: database --> - wow !!?!!

How many queries per page is recommended?

Thanks!

Aleks

Wow indeed. Nearly 4,000 queries seems incredibly high to me. And 11 seconds is an outrageously long time.

Can you leave your home page cached? The cached results are perfectly acceptable, but the uncached ones are horrific. The individual page cache files are not really a problem in terms of CPU usage or response time afaik (in fact they can speed things up quite a bit, as you can see).

A 900k site cache is definitely too big in my opinion. Even with lots of snippets and plugins mine are usually less than 500k. Ideally I think you want to try to get it down to say 300k or less.

I would love to see that home page! What on earth is it doing to hit the database that many times?

I figured out the problem. We have several ditto calls on the front page, and every call’s parent was set to 0.
so imagine those 8 dittos basically going through all documents 8 times on the page. pretty stupid.

the home page is cached at all times, so the speed is acceptable, but i completely agree - the loading time and the number of queries is just outrageous.
now i have to figure out how to combine all dittos in one call and bring other snippets with the include.

As long as you never leave the home page uncached I guess it doesn’t matter. It’s definitely not "right", but it won’t actually affect anything except that first page load. I would fix it anyway just so that it’s no longer a potential issue, but don’t let it keep you awake at night.

Quote from: rthrash at Oct 12, 2007, 03:45 PM

MODx would be good for the basic parts of the site and more static marketing oriented information. I’d really recommend having a custom database implementation for all those records though. Snippets and Modules would be good for managing those items (I certainly would not make them into individual pages). For example in an ecommerce perspective, anything over 250-500 records I consider the threshold for doing a purpose-built application/implementation inside MODx, keeping the marketing bits and "view" pages in MODx and the data filtering in from the custom development work.

Got a little frustrated reading this. My site will contain 100,000+ articles and many other items and I am targeting millions of records in the future.  So it seems that I can use Modx as a base system and do some customized code. Take for this article example, I can probably creat just one document for them and then use snippet to load the actual document upon request, but in this senario how about buildin cache? and other goodies like ditto?

I choose Modx over other php frameworks (cakePHP, CI etc) for it’s CMS backbone and Framework flexibility. We are urgently putting up a pitch site so any thoughts on the best approach would be appreciated.