MODx limits on the number of documents

Sooner or later MODx must support large docs tree and dirs (30.000 plus). This days 5000 docs is nothing.

Quote from: MasterzDee at Oct 10, 2007, 01:49 PM

Sooner or later MODx must support large docs tree and dirs (30.000 plus). This days 5000 docs is nothing.

Because everyone wants a site with 30,000 pages? I’ve never even come close to building one with a hundred pages, at least where more than a dozen or so were actually ever worth reading. But seriously, IMHO, anything with that large a quantity of articles needs special attention, regardless of what product you are using.

In any case, MODx will support what it can based on a number of factors, mostly related to the limitations of PHP and the software/hardware environment it is running within, as well as the architecture. Several changes are coming in the new releases that should help you better organize documents into distinct sections that can live by themselves, but large numbers of articles in the same sections are still going to be an issue. This is why some content needs to be managed in ways other than the standard MODx web content management paradigm (i.e. documents in the tree).

Finally, feel free to make suggestions and feature requests for MODx all you want, but keep in mind just telling us that MODx must do something is likely to accomplish nothing.

I’ve gotta somehow be guessing this is very much Adsense related... no?

My biggest MODx website has around 1 200 documents, and it’s a BIG corporate website with two languages and LOTS of data... Most of the websites I run are between 80 and 500 docs. Most corporate websites won’t go past the 5 000 doc limit (unless they have thousands of products...but then they’re likely not to run something based on PHP and MySQL, you’re talking enterprise level there...).

I don’t see the point of trying to manage this much data without building a custom database, no matter what CMS you’re using you’ll face the limitations of its environment...

Quote from: OpenGeek at Sep 30, 2007, 05:06 PM

@ZAP:
You can not turn off this part of MODx caching; period. This is not partial page caching we are talking about. MODx would not work without the siteCache.idx.php and in the current architecture, all 250,000 documents would need several lines each in this file, plus all of the PHP source code in the site definition (snippets, plugins, modules) would also be in there. You would have a file so large that it would likely never execute on any PHP installation, and certainly not with any reasonable performance.

I’m wondering at what size does the siteCache.idx.php file tend to become a problem. We have one site now that is experiencing occasional 500 server errors, and the siteCache.idx.php is about 550k. I can probably reduce this quite a bit if I make the snippets and plugins into include files, but is this file size significant enough that I should try to reduce it?

Quote from: MasterzDee at Oct 10, 2007, 01:49 PM

Sooner or later MODx must support large docs tree and dirs (30.000 plus). This days 5000 docs is nothing.

If you’re trying to do an ecommerce or db-driven site incorrectly, yes, 5,000 is nothing.

If you’re not trying to make singular pages for every single thing, you’ll do fine. I work at UT, and our College of Ed. is converting their site to MODx 0.9.6.1, and we’ve had little-to-no problems. Our siteCache is at around 378k right now (we’ve got in about 560 pages, 1/5th or so of the total content), and runs smoothly on a test server that’s only a G4 running OS X. I can’t imagine how fast it will run when it hits the real servers.

All that said, 0.9.7 puts forth some real improvements in this area, both PHP-side and UI-side. Also, with the introduction of Contexts, it will prove quite easy to isolate sections of your sites into simply manageable parts.

If you’re running at 250,000 docs (or honestly anything more than 2,500), I do have to ask what you’re doing. It sounds to me like you should be writing some real app dev code to handle things, rather than trying to fit it all into a CMS. MODx is great - but it doesn’t replace true application development, yet.

Just an additional note regarding some of the caching improvements that are coming in MODx 0.9.7 to address performance and scalability against a graph of volume a site’s content repository might consist of:

• More modular cache files, loaded on demand; config is separate from the document map, which is divided into contexts, scripts and content are separate, etc.
• More control over what parts get cached and which parts don’t; even how/where it gets cached. You can even disable caching entirely; however, the database load at that point would be tremendous, unless you had...
• Database result set caching to address database load in general, with support for memcached or your own custom cache implementations. Individual result sets can even be cached indefinitely, for a specified number of seconds, or excluded from the cache via the new xPDO-powered API.
• More granular ability to control caching on any kind of content element (i.e. chunk, snippet, TV, plugin, module, template, etc.) via tags or API, i.e. force TVs not to be cached.

But, that’s the challenge in developing any web application with an eye for performance and scalability; it’s a constant struggle to balance the two because they are, and will always be, at odds.

Quote from: splittingred at Oct 11, 2007, 11:15 PM

Quote from: MasterzDee at Oct 10, 2007, 01:49 PM

Sooner or later MODx must support large docs tree and dirs (30.000 plus). This days 5000 docs is nothing.

If you’re trying to do an ecommerce or db-driven site incorrectly, yes, 5,000 is nothing.

If you’re not trying to make singular pages for every single thing, you’ll do fine. I work at UT, and our College of Ed. is converting their site to MODx 0.9.6.1, and we’ve had little-to-no problems. Our siteCache is at around 378k right now (we’ve got in about 560 pages, 1/5th or so of the total content), and runs smoothly on a test server that’s only a G4 running OS X. I can’t imagine how fast it will run when it hits the real servers.

All that said, 0.9.7 puts forth some real improvements in this area, both PHP-side and UI-side. Also, with the introduction of Contexts, it will prove quite easy to isolate sections of your sites into simply manageable parts.

If you’re running at 250,000 docs (or honestly anything more than 2,500), I do have to ask what you’re doing. It sounds to me like you should be writing some real app dev code to handle things, rather than trying to fit it all into a CMS. MODx is great - but it doesn’t replace true application development, yet.

I’am running Entertainment site (celebs, movies, reviews, lyrics...etc) and in db I have 24.000 names, and each name have a lot of data (news, film, bio, pics, movies ...etc) , lyrics contain 30.000 artists and over 600k lyrics, db size is 3 GB. So I was thinking it would be good to convert site to MODx system, because now I do programming myself and is a lot of work and time consuming.

MODx would be good for the basic parts of the site and more static marketing oriented information. I’d really recommend having a custom database implementation for all those records though. Snippets and Modules would be good for managing those items (I certainly would not make them into individual pages). For example in an ecommerce perspective, anything over 250-500 records I consider the threshold for doing a purpose-built application/implementation inside MODx, keeping the marketing bits and "view" pages in MODx and the data filtering in from the custom development work.

Do unpublished pages count toward the ’maximum’ page limit?
Are there any tricks to reduce the cache size?
if we dont use the internal search - does it help with performance?

thanks!