I’ve hacked my modx core to not cache document map, aliases, document listing and content types in siteCache.idx currently stored in:
$a = &$this->aliasListing;
$d = &$this->documentListing;
$m = &$this->documentMap;
$c = &$this->contentTypes;
My core populates these arrays for the requested document fetching its document map, aliases, document listing and content types from the database and also fetches the same data for its parent and all its children.
I haven’t yet come across a need to get any more document data than the document’s parent and children.
This massively speeds up loading with 400,000 documents!
I’m afraid my hacked code is a real mess and I can’t remember what I’ve changed so I’m afraid I can’t post it here but may I suggest something similar is done to modx in the future release to make it more scaleable?
Quote from: ZAP at Sep 30, 2007, 06:59 PM
100MB is a huge file, no matter how you handle it. I think that I’d split up the XML and store it in a custom MySQL table (properly indexed) and use a snippet to display the data as desired. I would think that would work with the current MODx core, and depending upon how well you optimize your table I imagine it will behave well enough. As long as you’re not outputting huge files I would think you could cache them, but you may or may not want to do that anyway. As I understand it, this should keep the MODx sitecache file from containing your XML data.
What you lose is the ability to search through this data using a standard MODx snippet, since those aren’t designed to index dynamic content or data in non-standard tables. But writing your own search snippet shouldn’t be all that tough, given that your data is standardized and MySQL will pretty much do the work for you. And of course you don’t get any of the other features of using the MODx system, so for example you may need to create a separate module to import or edit your XML data (since you won’t be able to do this via the Manager).
You’d still benefit from MODx’s templating system, API, etc., so I guess I’d give it a shot in MODx and see what happens.