Answered Is there any way to increase the number of Resources?

ModX uses the 'site_content' table to store Resources. The primary key on this table is an Int(10) leaving about 4 billion resources to be shared amongst all contexts. In most situations this is absolutely ideal. I, however, would like to increase that number... What I essentially want is the 'site_content' table to be duplicated 3 times (1 for each context), but still in the same ModX install. Is there any way to do this without changing the back-end or writing a CMP?

As far as I can tell, I would have to write a CMP and a couple of Custom Resources in order to make this work. I need to know if there is an easier way. (Like simply creating a Context Setting that tells what table to use?)

This question is ultimately about scalability rather than immediate need, but I figure whatever I can do now easily will limit what I have to change when the site is active with users.

Note: It would be even better if it were a BigInt, but I know that PHP has limited to no core support for bigints, and I do not want to adjust the ModX source. [ed. note: fuzzicallogic last edited this post 14 years, 2 months ago.]

Do you forsee having more than 700 million resources in each context?



------------------------------------------------------------------------------------------
PLEASE, PLEASE specify the version of MODX you are using.
MODX info for everyone: http://bobsguides.com/modx.html

Yes... but not right away. It has to do with the type of site I am creating...

More Details: It's a networked communications blog for developers, teams and their projects. I'm not including micro-blogging, but it will have examples, tutorials, news, etc for each context. There's more to it than that, but those are the basics. Comments, themselves, are being handled by Quip, so I don't have to worry about those.

Assumption: If I assume 1000 users and 1000 projects, then there can only be 4000 posts per. Add teams into the mix and the number drops dramatically. If I assume 1/10 (100 teams), then its about 40 per. These numbers are very specific for demonstration.

The Reality: Since a single developer is likely to create more than one project, but are not guaranteed to join a team, the relationships are more likely to be X users, 3*X projects, and .1*X teams. Its certainly not going to be quick, but if any of those numbers goes up, the number of resources available for them will drop like a rock. 1000 users is a long way away, but I want to plan so that I don't have to break it entirely later.

Note: In Revo 2.2, the key is an unsigned int(10), so that's a little over a 1.3 billion per context.

It sounds like the scale of what you're wanting to achieve is going to require either modifying the core and database to physically allow a higher number of primary keys, or building a new blogging component from scratch. You're going to have to find a very solid, scalable caching solution. I've never even considered that situation so am not sure what to expect.

I don't know how Revo would hold up with that many resources, have you tried populating the database programmatically with a huge number of resources? What happens when you load a page in the front end, how long does it take to load? What happens to the resource tree when you log in to the manager?

Back to your point about The Reality. I wouldn't worry too much about the possibility of running into so many resources at this stage. My philosophy is 'get it live now, worry about scaling when the time comes'. I'm not alone there. You could spend a huge amount of time preparing for a scenario that might not arrive. If you do find yourself approaching that situation, then presumably you'll have a very successful business going and will be able to afford the engineering resources to allow the site to continue to scale.

Revo holds up fine with 1,000s of records... The ModX core is amazingly fast, if utilized correctly. and I'm not having caching issues right now (even despite the fact that I'm accessing 400+ resources per page load, it only takes 1sec to load the page). I haven't modified the core and would like to avoid it (if possible). Doing so might slow down the core processing or infringe upon other functionality.

I understand exactly what you are saying about a scalable custom blogging solution. I was just hoping that there was a hidden system setting to change the table from 'site_content' for just that context. This would solve the problem toot-sweet!

My philosophy is 'get it live now, worry about scaling when the time comes'. I'm not alone there.

In general, I agree with this philosophy. But, I also agree that if you can be scalable with little effort, you should probably do so. This does not look like that sort of circumstance, so I'll probably just have to deal with it for a while (which I am certainly okay with)

Quote from: fuzzicallogic at Jul 29, 2012, 11:01 PM

I'm accessing 400+ resources per page load, it only takes 1sec to load the page).

Just out of curiosity - how/why are you accessing over 400 resources in a single page load?

I understand exactly what you are saying about a scalable custom blogging solution. I was just hoping that there was a hidden system setting to change the table from 'site_content' for just that context. This would solve the problem toot-sweet!

There's no system setting unfortunately. I think your best bet might be to look at extending Articles, as that will at least provide a way to manage the resources in the back end. You might even be able to save your 'Articles' into additional site_content tables and sort out some kind of request handler to display them in the front end - it depends just how much of the default resource functionality you want to preserve. It's not going to happen without a lot of custom coding - nothing I know of will do it out of the box

I would imagine that a news or blog site could easily be accessing hundreds of resources if it's listing several categories in one section of the page, as well as listing archives. Then the footer might contain blocks that are taken from yet another group of resources. Add sidebars, perhaps for the archives as well as other aggregated content. Run this for a year or two on a busy site, and you'll be using getResources and Wayfinder and other aggregation and listing snippets to sort through hundreds if not thousands of resources.

Another scenario might be a support site for a large hardware distributor. You can easily have thousands of pages of support information for all of its products ranging back several years. Listing these in select drop-downs could get quite intensive. Although a lot of AJAX would help reduce the load in such a case. And I would be inclined to use a custom table to cross-match model numbers with the relevant resource for search and simple listing and linking purposes. Or maybe a MIGxDB TV... hmmm.

Resources carry a lot of overhead that you might not need and MODX does a lot of parsing of things like pub_date, unpub_date, etc. when loading, saving, and preRendering that might be unnecessary.

A custom DB table (with enough capacity in the fields) and only the fields you need might be a better option.



------------------------------------------------------------------------------------------
PLEASE, PLEASE specify the version of MODX you are using.
MODX info for everyone: http://bobsguides.com/modx.html