Document / Cache Limits

So awhile back I read about the tests with ModX and the issues with the cache becoming to large after maybe 10K documents. My question is, can this limitation be overcome by turning off the cache for every document in ModX? If it solves the cache overload issue, what other issues would pop up in it’s place?

Quote from: JeremyL at Nov 15, 2006, 03:46 PM

So awhile back I read about the tests with ModX and the issues with the cache becoming to large after maybe 10K documents. My question is, can this limitation be overcome by turning off the cache for every document in ModX? If it solves the cache overload issue, what other issues would pop up in it’s place?

No, this does not solve it. The partial document caching is not the issue, but rather the site cache, which contains a list of all the documents in your site with some important metadata, as well as the source for plugins, snippets and chunks. Many things are being done in the infrastructure to remedy this moving forward, but for now there is no "easy" solution to overcoming these barriers.

This is a substantial issue for a couple projects I have in the pipe as they will be collecting news stories over a long period of time and this could result in a HUGE number of documents. Since these documents could be archived in some 3rd system somewhere, isn’t this a possibility? Just a utility that sweeps a set of files by date or whatever into and archive table that is served up in some other manner - even if that is external to the MODx architecture. It would be a band-aid, but should do the trick, no? Is this possible?

Quote from: howeydesign at Nov 24, 2006, 10:21 PM

This is a substantial issue for a couple projects I have in the pipe as they will be collecting news stories over a long period of time and this could result in a HUGE number of documents. Since these documents could be archived in some 3rd system somewhere, isn’t this a possibility? Just a utility that sweeps a set of files by date or whatever into and archive table that is served up in some other manner - even if that is external to the MODx architecture. It would be a band-aid, but should do the trick, no? Is this possible?

You could absolutely use a custom news system. Just author or integrate one and MODx only has to manage your News page and maybe a separate Archive page; everything else is handled dynamically and without the MODx document caching issue.

However, these issues will be improved and a much greater threshold of documents will be possible, probably before they run into a problem with document count...unless they are publishing 100 pages or more a day.