MODx limits on the number of documents

Quote from: DiegoBM at Jun 06, 2010, 07:08 AM

May I supose that uncached documents don’t take into account for this document limit guideline (even though they remain in the database maybe causing other side effects, but those could be solved optimizing/filtering my database requests I guess)?

No; whether a Resource has partial-page caching enabled has nothing to do with the site cache, which is where the "limitation" is centered. I always use partial page caching and make specific Elements non-cacheable where necessary. IMO, partial page caching is one of the most important aspects of building a scalable site with MODx (or any PHP application for that matter).

Quote from: DiegoBM at Jun 06, 2010, 07:08 AM

For example, supose that I’m running a news site that is reaching the 5K "limit". If I eventually unchache (and unpublish) older entries (but maintain them for record purposes) and clear the cache, in order to consistenly maintain a number of 3K cached/published documents. Should I worry about this limit and performance decrease (assuming that I have optimized my DB queries and tables)?

Unpublishing does not solve the problem either, because these records are still in the site cache so that manager users that have permission to view unpublished pages can do so.

Again, and this is my opinion, if you have more than 100 of anything that is identifiable as something other than just a web page (products, news stories, etc.), I consider it data and something that should be managed separately from a MODx Resource. A Resource is a unique View, or way to present some specific type or piece of data in a web page. MODx is great for constructing these Views, but you really shouldn’t create the same View for every piece of data that is in the same format unless the quantity is practical and is expected to remain that way (I say up to 100 Resources is practical; the "limit" for common environments is estimated at around 5000 MODx Resources).

I’m not sure I totally understand having the data separate from the view (or I do in a non cms environment.) how would you separate the data in modx?

Also has caching been changed in 2.0?

Basically, it just means that instead of having the information stored in the fields of each document, you’d create a custom DB table to hold it all. You have stored all the data, but created no resources (with all their unnecessary overhead).

Then you’d create a snippet (or snippets) to display the data. Ideally, you’d create Tpl chunks with placeholders in them to do the actual formatting. The snippet receives the names of one or more Tpl chunks as parameters, retrieves them, queries the DB, and iterates through the records, showing each one by replacing (or setting) the placeholders in the Tpl chunk.

Of course in some cases, the query might only get one record.

> Jason wrote:.
> Again, and this is my opinion, if you have more than 100 of anything that is identifiable as something other than just a web page (products, news stories, etc.), I consider it data and something that should be managed separately from a MODx Resource. A Resource is a unique View, or way to present some specific type or piece of data in a web page. MODx is great for constructing these Views, but you really shouldn’t create the same View for every piece of data that is in the same format unless the quantity is practical and is expected to remain that way (I say up to 100 Resources is practical; the "limit" for common environments is estimated at around 5000 MODx Resources).

======

I work for a company that publishes many magazines that have way over 5,000 articles. (Tens of thousands). The editors post them via a standard web interface, via a CMS. Some are in Ektron, which stinks; some are in Nstein, which I also think is vastly over complicated. The Ektron ones are moving to DotNetNuke. I’m pushing MODx for some of the smaller sites (with a few hundred articles). Prior to their DotNetNuke decision, I was pushing MODx for the bigger sites, but there’s that pesky 5,000 limit.

When you say 100 resources, are you talking about 100 articles, i.e. web pages? Or are you saying that one article can be taking up more than one resource, i.e three or four chunks, some snippets, etc? For example, does a home page, with many blocks of ditto snippets, count as one resource?

My own personal web page, which is now using MODx Evo 1.0.3 (and I’m loving it) is at http://significatojournal.com. It has around 400+ articles.

I’m a wee bit confused, since I’m coming from Perl, and am not intimately familiar with PHP caching. I’m hoping MODx Evo can handle at least the 5,000 articles.

I’ve also run into an issue with 1.0.2 (on a client site) and 1.0.3 on my site, where all of a sudden the pages returned a parse error, which was cleared up by going into the manager. Is this a problem of the site cache getting corrupted? (Maybe I’m mixing issues here.) I’m thinking maybe I should run a cronjob to delete the site cache every so often.

Thanks,

Peter

You should be working with Revolution, then. It doesn’t work the same way Evo does, and doesn’t have the limitations.

This has nothing to do with Evo or Revo and everything to do with having a view of data in a common format vs. making every piece of data a complete view in of itself. MODx is great for creating dynamic views. The reason WordPress can handle any number of posts is because it does not treat every post as a separate web page to be managed and cached separately. It’s literally a single theme in WordPress that presents data from a standard format. A MODx Resource should represent a theme in WordPress when it comes to articles or other content that may be presented in great quantities. It should not represent each article as a Resource. That’s like duplicating the code of a WordPress theme for every article you write IMO.

MODx is a framework that can allow you to make simple web pages or complex dynamic web views like I am describing/recommending when managing a large number of content artifacts. Just because you can do something (i.e. create every article as a MODx Resource) doesn’t mean you should.

Dear Jason,

So, are you saying that when one starts plugging articles into the MODx content manager tree, with the Create Resource Here button, that that’s *not* the right way to do it?

The manager is *the* interface to creating resources, so if we have to create a whole new interface to a separate database of articles, it removes the convenience and value of MODx, with all of its attendant cool things, like TVs, etc.

I’ve got a content tree with nine major categories (i.e. top menus), each having a certain number of sub-sections. Each section contains a variety of articles. It’s a typical magazine structure.

I’m a bit confused as to how you would accomplish the same thing. Also, is one article a resource, even if it pulls in chunks, etc? Is a section page, with a ditto call, one resource?

So far, I’ve built 4 sites in MODx Evo 1.0.3, all based on the concept that it could handle 5,000 "resources", i.e. articles, section pages, etc.

I think we need to get a solid handle on the real limitations (in terms of quantity) of MODx Evo.

=> added comment:

I built my own CMS, in Perl and MySQL, which easily handled 20,000 articles, before I switched to another CMS (for the company I work for). I realize that it didn’t have the same architecture - it wrote each article out as a static html file, and used file includes for menus, etc.

My point is that MODx has a really great wiring system, with snippets, chunks, TVs, etc. I’m attracted to it because of its power, ease of use and just plain loveliness .

If each article out of 5,000 shouldn’t be a resource, then it looks like (to me) we’d have to abandon the above wiring scheme, and interface, which doesn’t look like good news.

Best regards,

Peter

I think what Jason is saying is that you can certainly do it with each article as a resource, but there is a price to pay in speed, efficiency, and server resources used and another price to pay in terms of the design goal of separating the view from the data.

A resource comes with a lot of overhead in both storage and processing, especially in Evolution. It has many fields, many of which you aren’t using, and it is processed by the parser to handle any potential tags, TVs, permissions, etc. whether those exist or not. If there are no TVs, no tags in the content section, and all you need is the title, creation date, author, and content, all that extra processing is unnecessary. It’s much more efficient to create a new table with just the fields you need and a Tpl chunk to display them the way you want and you only need one resource for the display and another for the user to enter the data. When the number of "documents" gets very large, that efficiency becomes more important.

Another way to look at it is that the view of the article you see in the Manager contains a lot of fields you aren’t using and you are saving that entire "view" for each article, rather than just the data you need.

OTOH, if you are using TVs, MODx tags, and most of the fields, then it’s worth paying the price, but you should probably move to Revolution where it’s all done much more efficiently.

Dear Bob,

Thanks...

Your post above implied that a user could enter data into a foreign table from the MODx interface. Is that correct?

I guess at this point, what threw me off was Jason’s mention of the max of 100, versus the 5,000. If MODx Evo can handle up to 5,000 articles (even 3-4,000), then for most clients it will work. But if things start to degrade after 100, then it’s quite a problem.

I’ve waited to move to Revo, because it wasn’t ready yet.

Also, it would really help to see an example of Jason’s paradigm, above, in terms of processing and presenting the foreign data. My main concern is the user interface to that other data. I could use my other CMS interface that I wrote, to insert and modify the data, but that creates a weird split, where MODx is only being used to present the data, and the other CMS is being used to edit the data. One might think, why use MODx at all, in that case?

Peter

Resources (web pages) are one thing, and they serve the purpose of organizing and managing what will become web pages very nicely. But if you only want to store and retrieve a couple fields for a number of items, it’s overkill. Better to use a custom table to store those items. I often use Jot for simple lists where resources and Ditto would work, but would be more complicated to manage and would be like using a shotgun to shoot a mosquito. Jot works great for lists like FAQ question/answer, restaurant menus, events, anything that will have only a few fields. Or you can use a custom table and a custom snippet (which is really all that Jot is) to filter, sort and display the items, and a simple module with a form to manage the table. And, if you really want to, you can use resources.