Is there a document limit to what MODx can handle

I was curious if anyone has used MODx to handle a large amount of documents. A VERY large amount. Say over 100,000.

We are a How-to info/e-commerce site and my boss asked me if we could create product information pages for all our ecommerce items. Sort of like how cnet.com has product review/infomation pages for different products.

We are usuing MODx for our articles, roughly 5,000 records in MODx, but I wanted to get someone elses opinion before I even attempt this. Do you guys think MODx can handle over 100,000 documents, with each one possibly having its own template variables? Or will this cause my server to explode - jk

LOL... wow.

You are treading new ground here for sure, but I’m very interested in watching how this works out... would certainly make for an awefully interesting case study. I suspect some careful custom tweaks to the queries and indexes would improve any results (and should probably become part of the core). Heck I don’t even know if the database (MySQL) is going to be the mitigating factor or not...

Well, of course your highest doc ID is 4294967295, so that’s 4,294,967,295..

You already have 100,000 products in an existing ecommerce setup, is that right?
And you already have 5,000 MODx articles?

You may need some sanity tweaks to the manager, but I’d say if your ecommerce is OK with 100,000 products, MODx will be OK too.

I have to chime in here; this has been discussed in other posts, but in general, if you have a dedicated server and lots of power, you could run MODx as is with 5000 to 20,000 documents and be ok, though the server won’t like it much. 5000 documents roughly means 10Mb or more of server memory used on every request, which is not good. This scalability limitation is due to the monolithic nature of the siteCache.idx.php file. This is a fundamental problem that I am working on for future releases, but I see no easy way to solve it at this point. I’m not saying it won’t handle it at all, but numbers over 5000 documents could get really hairy.

Here are some benchmarks on the siteCache file with various numbers of documents for reference. These may not be accurate for every site with the same number of documents, but should help you get a better feel for the practical limitations...

Thanks for the input guys.

I was worried that the whole caching situation would be an issue when using that many documents. We do use a dedicated server for our articles, although it is not live to the public yet. So it is still to be seen how MODx will handle 5,000+ articles with a HUGE amount of requests per day. It seems to me that adding all those product pages would probably be too much with how the system is currently set up. But, we are still definitly using MODx for our articles.

My situation is this, our articles are currently built in with our e-commerce platform hosted by a third party. This system is sluggish and un-tweakable, so I was asked to find a new CMS to host our content. After weeks of research and testing, I chose to use MODx because its level of flexibilty - and I really like the manager interface.

Hopefully, if all goes as planned, we will be "flipping the switch" to our MODx site by next week. So I’ll keep you guys posted on how well it performs with 5,000+ documents on high traffic site.

Thanks again.

I was linking that if your ecommerce system works then MODx would work fine. Your ecommerce system is sluggish... actually I’m not sure whether you’re saying that MODx will be on the same server or not.

Quote from: DIYmodder at Sep 22, 2006, 01:01 PM

It seems to me that adding all those product pages would probably be too much with how the system is currently set up.

Another option, if MODx’s document caching model gets in the way, would be to store the product information in a second database and use custom code. Pages in MODx don’t have to be based on MODx documents, you can get data from other sources.

--

Having read the thread Jason linked to, which contains alarming stats from a false test in 0.9.2.1, it would be an interesting exercise to repeat those tests with 0.9.5 once we have the default content sorted. The tester shaved a third off the MySQL times by using the caching properly.

I have been following this thread with great interrest, not only from a marketing point of view but also because a prospect of mine will require a system to handle a lot of documents too (several thousands, exact number TBD), and when I talked about MODx I got this very question. I am busy with two project which will soon go live, but as soon as it’s over, this will be the next order of business for me.

But back on testing. There had been a benchmark posted on opensourcecms too, about MODx performance. The problem with stats is that doing real stats is not trivial and most of the time people who do this kind of testing do not master the testing environnement or the methodology. I am by no means an expert on this particular subject, but as a business process consultant I have seen enough quality test to know you can’t base your results on a few samples, you have to be in control of testing conditions, follow a strict procedure and do enough samples, draw statistics from it (especially true in complex environnements)... it’s really a full time job. Until we have some consistent results, I won’t draw any conclusion from those.

I am eager to get feedback from you, DIYmoder, nice MODx project you have there : undoubtedly a potentially very interresting case study, as Ryan said

To answer Paul’s question:

Your ecommerce system is sluggish... actually I’m not sure whether you’re saying that MODx will be on the same server or not.

MODx will not be on the same server as our "slugish" ecommerce platform. In fact we may actually be adding more resources to our already dedicated server, just for MODx. So I’m hoping that serving 5,000+ documents will work out allright. The MODx manager has been working fine in our test environment, while about 6 or 7 editors have been editing articles in there for the past week. I think by next mid-week we will start redirecting users to the MODx server for all our article content.

As for my large amount of product info pages, I’m thinking I will go ahead and try to insert the data into a local test server. From there I can see how the manager handles it and also see the strain on the server CPU and memory. Unfortunatly, I’m not an expert on benchmark testing and all that, so I won’t be able to provide thorough testing results like in the post linked by OpenGeek, but I can give you an overall idea of what I see. More to follow next week, I’ll keep you posted.

Hi guys,

Just thought I’d give an update on my progress. In the end of my import, I had a little over 90,000 records in MODx, only about 13,000 had any template variables associated with them. Unfortunatly, I don’t see something of this size working in a current version of MODx. The problem is the caching system, as OpenGeek said it would be. I was able to browse around the manager with all the documents in the menu tree, but refreshing the cache was not happening. I set my php max_execution_time to 120s and it still would not complete. I thought about changing how the cache processor worked, to exclude all these documents, but so much of the document parser depends on this - for alias links and just generating a page. So it seems I will have to take a different route to solve this. Probably something to the effect of what PaulGregory recommended.

On the plus side, I left all my product index pages in the system, making the record total 13,891 and MODx seems to handle this amount fine. So if your looking to host a couple thousand docs in MODx, I’d say go for it. Of course, this is just testing on my local machine - so make sure your web server is powerful enough to handle it.

Still with this many docs, my siteCache.idx.php file is 2,395KB in size - pretty hefty, but pages still seem to load relativly quick. After about 30 min. of modifcations to cache_sync.class.processor and document.parser.class.inc, I was able to trim this cache file down to 1,558KB and reduced total page load time by half a second (got times by using MODx time variables). There is a lot of redundant data in that cache file. I basically combined all those doc arrays ($a, $m, $d, $c) into one array - of course, that messes up a whole boat load of things in the document parser, but is easily adjusted. This would not be of concern for someone with a smaller amount of documents, but if MODx ever wants to support larger sites with large amounts of data, something will have to be done with the cache processor. Just my 2 cents.

Anyway, I’m still going to use MODx for our 5,000+ articles, still the best open source CMS I’ve seen.

BTW - MODx 0.9.5 Beta looks SWEET, good job guys.

Nice feedback there, much appreciated !