MODx limits on the number of documents

Quote from: peterbrown at Jun 30, 2010, 03:05 PM

Your post above implied that a user could enter data into a foreign table from the MODx interface. Is that correct?

What I meant was that you’d create a front-end form or a custom Manager page and use the MODx DBAPI to store the data from the form in an extra table in the MODx database. Sorry if that wasn’t clear.

I guess at this point, what threw me off was Jason’s mention of the max of 100, versus the 5,000. If MODx Evo can handle up to 5,000 articles (even 3-4,000), then for most clients it will work. But if things start to degrade after 100, then it’s quite a problem.

I’ve waited to move to Revo, because it wasn’t ready yet.

The degradation would certainly be measurable after 100 docs (or less), but I doubt if any user would notice it until many more than that.

Revo is remarkably stable. It’s in RC2 now and RC3 should be released shortly. The remaining bugs are in seldom used parts or are mostly cosmetic (e.g. display problems in IE7). Many sites (including this one) have been using Revolution for some time with no real issues and no downtime.

Dear Susan and Bob,

Thanks for your feedback. I appreciate it.

I guess the reality is that I’m using many of the default fields, with an additional significant number of TVs, in the magazine site I’ve created. Here’s a sample article:

http://significatojournal.com/columns/back-porch/hair-loss-and-world-peace

Here’s a list of TVs that I use:

- article_type (for formatting of special article types, like short stories)
- content_tags (like "attributes", for filtering in ditto)
- headline_thumbnail (for displaying thumbnails on section pages)
- metatag_keywords
- newsletter_volume_number (to include the article in a newsletter)
- section_footer
- sitemap_exclude (a ’1’ means don’t include it)
- static_author (for articles where there’s no writer in the external table)
- static_author_attribution
- subtitle
- writer_id
- code_block1
- code_block2
- code_block3
- code_block4
- code_block5
- code_block6
- code_block7

’writer_id’ for example, stores an id number that a snippet then looks up against an external table, and pulls in the author’s byline, photo, email and attribution.

code_blocks are used for easily posting things that TinyMCE chokes on, like Amazon code to sell a book, or YouTube embedded video, etc.

I also use chunks quite heavily, as shorthand for items, e.g. {{wiki}} to display a string like "Images from Wikimedia Commons".

If I had to use an external table for the articles, with all their attendant TV fields, etc, then it would be almost like moving back to the CMS that I wrote, that I *just* ported all these articles from, into MODx.

I’m wondering where the number of 5,000 came from? With a magazine site like mine, I’d hate to see a significant slowdown at the 1,000 mark. I’ve got 400+ now, and things are working fine (Evo 1.0.3).

If I move to Revolution, will I be able to keep the whiz-bang functionality of MODx, and still store each article as a resource, while breaking the 5,000 limit?

I may be confused, in terms of what MODx is supposed to be doing, but out of the box it seems to be a really great CMS, if one creates an article as a resource.

I’ve never used WordPress, but it seems, from Jason’s description, that the behind the scenes wiring is transparent to the user. That’s what users want, I believe.

To use a "Resource" as a template to present data from a foreign table seems to be jumping over the whole point of MODx, in terms of ease of use and all the great wiring that comes with it.

Has anyone ever created a magazine style interface in MODx while only using a resource as a template to external articles (per Jason’s concept above)?

Thanks for all your help on this!

Peter

The problem is in the site caching system. Every resource produces three arrays like this:

$this->aliasListing = array();
$a = &$this->aliasListing;
$d = &$this->documentListing;
$m = &$this->documentMap;
$d[’home’] = 1;
$a[1] = array(’id’ => 1, ’alias’ => ’home’, ’path’ => ’’);
$m[] = array(’0’ => ’1’);
$d[’testing-jot’] = 2;
$a[2] = array(’id’ => 2, ’alias’ => ’testing-jot’, ’path’ => ’’);
$m[] = array(’0’ => ’2’);

to form the aliasListing, documentListing and documentMap. Imagine 15,000 array elements (three arrays for each resource); and two of these are two-dimensional arrays, which take up much more memory. These arrays end up bloating not only the index file itself, taking longer and longer to parse it and load the arrays, but also eat up memory as these arrays get loaded with each resource load. The cache file also holds all chunks, templates, snippets, plugins as well as the system settings. So the more code in the database for these items (as opposed to a few lines and including external .php files for the rest of the code) the bigger the cache file gets.

If I had to use an external table for the articles, with all their attendant TV fields, etc, then it would be almost like moving back to the CMS that I wrote, that I *just* ported all these articles from, into MODx.

hm, good point. bit confused about that topic, too. fortunately i don’t build sites that large (yet)

One of the great advantages of TVs is that they are not represented in the cache file!

With ManagerManager, you can create another tab to hold your TVs, making creating and editing resources more organized.

in revo we where able to 15000 recources each with 20 populated tv’s it made a single pageload take up to 3 seconds because of the big tv table (in one test we had more that 3.000.000 rows in there and the table was 800MB+) the modx tree did not load any more but the pages still worked, so I think the problem could be mysql when sites get to big.

Quote from: OpenGeek at Jun 29, 2010, 09:00 PM

This has nothing to do with Evo or Revo and everything to do with having a view of data in a common format vs. making every piece of data a complete view in of itself. MODx is great for creating dynamic views. The reason WordPress can handle any number of posts is because it does not treat every post as a separate web page to be managed and cached separately. It’s literally a single theme in WordPress that presents data from a standard format. A MODx Resource should represent a theme in WordPress when it comes to articles or other content that may be presented in great quantities. It should not represent each article as a Resource. That’s like duplicating the code of a WordPress theme for every article you write IMO.

MODx is a framework that can allow you to make simple web pages or complex dynamic web views like I am describing/recommending when managing a large number of content artifacts. Just because you can do something (i.e. create every article as a MODx Resource) doesn’t mean you should.

the problem when using a separate table is that you can’t use the nice resource API any more including all snippets that are made for it like ditto wayfinder etc.
we (B03tz en I) are working on a modx native web-shop component in Revo and our biggest discussion was : do we use a custom table for the products including the options or do we use modX resources and tv’s for that?

we still think that in Revo we like to use the resources because if we do that, we can use all the nice snippets to make menu’s etc use the tags in the templates lots of benefits come from the resources and the complete integration in modX, a custom table would make al these options unusable.

Dimmy

I’ll say it one more time: I recommend you DO NOT use Resources to represent data that needs multi-faceted views. Your should never have thousands of Resources, as these are supposed to represent distinct, dynamic views in your site. The perception that using custom data tables to populate your views does not allow the use of Wayfinder or Ditto is IMO irrelevant, as you can (at least in Revo) very quickly create views from custom data tables. TV’s and Resources are for constructing layouts; IOW for presentation. Not for data.

The whole benefit of MODx is being able to quickly write custom snippets to accomplish all the cool dynamic things you would normally do in a single PHP page. Modeling your data using MODx Resources and Template Variables simply because you want to use Ditto or getResources or Wayfinder is IMO a serious flaw in perspective when using MODx for building professional sites. It only makes sense if your data sets are extremely limited, which is great for very small endeavors with limited budgets.

And MySQL and the number of TV’s have nothing to do with the site slowness, unless you are sorting on TV fields or something (which I won’t even get started on). Having 15000 Resources in your cache file absolutely does though.

I think there is a middle solution. You could use modx resources to easily proof the concept of your application and have something running in a few days. If you’re expecting thousands of resources to use the same template, you must start coding your application as a modx module ASAP. When the module is done move the data using an export script

Quote from: OpenGeek at Jul 08, 2010, 01:18 PM

Modeling your data using MODx Resources and Template Variables simply because you want to use Ditto or getResources or Wayfinder is IMO a serious flaw in perspective when using MODx for building professional sites. It only makes sense if your data sets are extremely limited, which is great for very small endeavors with limited budgets.

Dear Jason,

I think the problem here is that MODx has been presented with a very slick interface to use the content tree to create resources, with TVs, snippets, chunks etc. Thus, when someone like me comes along (a Perl programmer who wrote his own CMS), the impression is that MODx will work out of the box as a full-fledged CMS for any type of site. I noted the "5,000" max on Evo, but was willing to live with it until Revo, which supposedly broke that barrier.

I’ll bet you a box of donuts that 99% of MODx users are under the impression that they should be using resources for every article. This is a marketing issue for MODx, because now you’re saying (if I hear you right) that one should not use MODx this way, but instead, we have to go and customize a large and significant number of coding mods just to *really* handle large document sets. You mentioned 100 resources as a max above.

So... even though I’m becoming more comfortable with PHP, I’m now left in a real pickle. I spent *weeks* migrating my site from my custom CMS to MODx. I’ve already built two sites for clients - one a magazine site that is already getting close to the 1,000 mark.

My choices seem to be:

- use MODx with resources and wait for it to bomb out
- spend *many* hours creating a "real" CMS, built on top of MODx
- move on to some other program that doesn’t have these limitations

None of those options are happy ones. I really like MODx!

Based on your comments above, I truly think you have to revise your positioning of MODx, and clarify right on your home page that it won’t deal with large sets of resources without significant custom coding.

And... based on that - if you want MODx to be THE CMS for people to use, then why not build the custom CMS solution you mention above, and make it part of MODx? Why should every user have to reinvent the wheel?

In my humble opinion, this may be a real deal-breaker for MODx unless it’s fixed.

Sadly....

Peter

I think in reality for a site with so many pages, there has to be a realisation that work needs to be put in beyond what you get out of the box to really make something optimised to serve thousands of pages. What MODx does is save you time with that process.

When we built the-wine-guide.com, we expected to need to cater for around 10 - 20k wines being entered into the database. Sooner or later, we’re expecting to have 20k pages on that site. I immediately ruled out using resources for this, simply because *in the long run* life would be so much more difficult dealing with a database structure that is in no way tailored to wines (think of the search queries!). It makes no sense (beyond immediate ease) to try to shoehorn a ’generic’ resource into something that could be hugely optimised by being its’ own data structure. Sure, you need to create the CRUD yourself but MODx gives you all the tools to do that in a way that is much simpler than starting from scratch - seriously, extJS and the whole connectors/processors design is amazing to work with.

In this case we only used *one* MODx resource for displaying each and every wine on the site, for all the reasons Jason mentioned above and more. Essentially you use a resource as a controller to display data from your custom tables. With Revo it’s really not hard to do that, especially with xPDO where your custom data can be loaded as simple as:

$wine_id = $_GET[’wine’];
$wine = $modx->getObject(’wine’, $wine_id);

In Evo it wouldn’t be much harder to retrieve data either, just a few more lines of SQL.

That being said, it’s fine to use resources as pages on smaller sites (brochures, blogs) and I’ll agree that 99% of MODx’ers do this, but that’s because 99% of sites out there simply don’t have the requirement to serve 1000’s of pages. When you’re talking about magazine sites with 1000’s of pages, I think it’d be smart to leverage and built on top of a framework like MODx for all its’ worth - that’s necessarily going to require some ’customisation’ work to get the best performance (a budget, if you will). That doesn’t mean a hack to the core or anything like that - it means that for exceptional requirements, some custom work will have to be done using the framework.

I actually think @cipa’s recommended middle ground is a great idea - using TVs and resources does allow for rapid prototyping, but it’s in no way a replacement for custom tables when you’re talking about 1000’s of pages.