MODx limits on the number of documents

You can always turn caching off completely in MODx and use a reverse proxy in front of Revo, from Wkipedia :-

Caching: A reverse proxy can offload the Web servers by caching static content, such as images, as well as dynamic content, such as a HTML-page rendered by a content management system. Proxy caches of this sort can often satisfy a considerable amount of website requests, greatly reducing the load on the central web server; another term is Web accelerator. This technique is also used for the Wikipedia servers.

This is more normally used in load balancing scenarios admittedly but there’s no reason why this can’t be employed, you’re then limited by how fast you can render a page when you need to.

Quote from: OpenGeek at Jul 08, 2010, 01:18 PM

I’ll say it one more time: I recommend you DO NOT use Resources to represent data that needs multi-faceted views. Your should never have thousands of Resources, as these are supposed to represent distinct, dynamic views in your site. The perception that using custom data tables to populate your views does not allow the use of Wayfinder or Ditto is IMO irrelevant, as you can (at least in Revo) very quickly create views from custom data tables. TV’s and Resources are for constructing layouts; IOW for presentation. Not for data.

Thanks for that! This we can take with us while creating the webshop. Maybe we should rethink wether we should use resources...

Quote from: OpenGeek at Jul 08, 2010, 01:18 PM

The whole benefit of MODx is being able to quickly write custom snippets to accomplish all the cool dynamic things you would normally do in a single PHP page. Modeling your data using MODx Resources and Template Variables simply because you want to use Ditto or getResources or Wayfinder is IMO a serious flaw in perspective when using MODx for building professional sites. It only makes sense if your data sets are extremely limited, which is great for very small endeavors with limited budgets.

Ok, this makes sense...TV’s are thus, in fact created for presentation instead of keeping data. I have never thought of it this way...thanks again.

Quote from: OpenGeek at Jul 08, 2010, 01:18 PM

And MySQL and the number of TV’s have nothing to do with the site slowness, unless you are sorting on TV fields or something (which I won’t even get started on). Having 15000 Resources in your cache file absolutely does though.

If we were to use TV’s for product-data...we would of course need sorting on the TV’s...but i doubt wether this is the case after another meeting

Quote from: shamblett at Jul 09, 2010, 09:06 AM

You can always turn caching off completely in MODx and use a reverse proxy in front of Revo

But there’s alot of modifying to be done if a system like this where to have any use. If you
start caching complete pages instead of MODx’s specific element caches how will you know if
1 page’s dynamic content has changed?

You’d have to invalidate the cache some how...for a specific piece of code from a specific page maybe even.
In MODx this is all automated...but not when you bypass it and use another cache system in front of Revo

Agreed this is extra work no matter how you do it however Revo has been designed to be extendible.

In Revo you can replace the cache subsystem with your own, you could just do something like not caching docs at all, just picking up cache change requests and invalidating these in your reverse proxies cache for example(Varnish supports this). Whatever scheme you chose here is done once and will work on all your installations.

Or you could keep adding new tables/CRUD operations every time your customer changes his mind. Then if your next customer wants the same do you go through the whole process again?

Its a value call on which way to go true but if your going to write code anyway it may be worth considering.

You are so right...this is very frustrating. As now we have to choose a path which will probably
set course for the future (or at least a considerable amount of time).

Develop a means to be able to cache large amounts of data in resources and TV’s...or handle it all
in custom tables.

On the other hand, if one should keep the MODx design filosofy in mind whilst developing a webshop for instance (or any
other component for that matter) you can at least have it very modulair (am i saying that right?) so that most of the
crazy ideas coming from your customer are possible anyway.

It’s gets tougher if you’re not a real scripter though...if you can script "modx" but not PHP in general and Javascript...
Or do not know how to invoke database calls...then this is not possible and a caching system would be easier yet again...

Yes, if you work in the ’comfort zone’ of MODx with TV’s, Templates, Resources, Chunks et al implementing crazy ideas from your customer is a lot easier, so is prototyping, management etc.

Personally I’d rather do this than implement another scheme using custom tables and the rest to get round a caching problem, I’d rather spend the time fixing the caching problem and sharing the fix so other people can pick it up. Just because Revo is flexible enough to make custom table extensions possible doesn’t necessarily mean that’s always the way to go.

Makes sence...i’m actually waiting for OpenGeek to slaughter one of our comments. I have a feeling he
wouldn’t agree with this filosofy.

I’ve been thinking all day about what to do...for the webshop’s purpose. I’ve also read up on some other
caching systems (3rd party) and while there are some nice solutions...there’s not a complete solution yet.

I’m actually thinking of writing one from scratch in my spare time...

Dear All,

I think my frustration with this issue is that I spent a boat load of time looking for a CMS to migrate to, away from my custom Perl CMS (which had no document limitation pe se).

I settled on MODx, assuming that 5,000 resources (as documented) were ok, and that that limit would be broken in Revo.

Rather than having to recode a CMS on top of MODx, I’ve been hoping to find a CMS that handles large document sets out of the box.

I think that if MODx wants to position itself as an industrial strength CMS, the team should solicit code contributions from folks like HK that have already done some of this add-on coding, and then come up with a CMS solution that actually works for large document sets, and still has all the bells and whistles that one would expect - much like the default MODx manager.

I realize that MODx is billed as a "An Open Source Content Management Framework", but as I’ve said before, I believe the limitation on how many documents are actually usable needs to be strongly communicated. 5,000 doesn’t seem to be accurate.

I think MODx *should* be an industrial strength CMS, able to handle hundreds of thousands of documents (or even tens of thousands). But can it be, and will it become that? Is that in the MODx team’s roadmap?

Peter

I can’t speak for the MODx team...but i don’t think this is currently on the road map since it is a framework and someone with that high a demands has the resources to build something on top of this powerfull framework.

At least...that’s what i think

Quote from: hk_modx at Jul 09, 2010, 02:48 AM

I just read through some of the other users’ thoughts - I’ll agree it sucks that developing sites with 1000’s of pages isn’t easier, but I would never approach a project of that scale with expectations that it would be, I just personally think it’s unreasonable to expect to not have to create any custom snippets if you’re building something of significant scale, although I can see how by default you might try to do so and hope for the best.

Quote from: shamblett at Jul 09, 2010, 01:38 PM

Personally I’d rather do this than implement another scheme using custom tables and the rest to get round a caching problem, I’d rather spend the time fixing the caching problem and sharing the fix so other people can pick it up. Just because Revo is flexible enough to make custom table extensions possible doesn’t necessarily mean that’s always the way to go.

I agree with both hk (James?) and shamblett.

1. with hk, If you are going to build a really huge site, extra work will be involved. (example: extra dedicated server, custom development, etc.)
2. with shamblett, i would rather focus on letting modx handle infinite resources.

Here’s some things I have done to do this.
1. on plugin docSave/edit/, etc - save the relevant documentObject fields to another table. Then use a custom snippet to show it
2. on plugin docSave/edit/, etc - save certain objects into a lucene index. The use onDocNotFound to run a snippet to show the results on the frontend