I thought that with Revolution out the Evo version was pretty much a dead project. But to my surprise the development apparently continues and the latest few releases have been amazing. On one of my sites based on Evo, I have been struggling with the "5000 documents limit" for a long time and I was forced to come up with several solutions to prolong the life of the website. And now I have decided to go public with them since there is a chance that they might be useful to someone. You can check the site on http://www.svoboda.info. If you display the HTML source code of the page, you can see down at the bottom the loading times. The site runs on a standard shared webhosting. Since my solution is mainly about tweaking the core, I will try to explain it as much as possible.
The 5000 documents limit
The most important cache file in Modx Evo is the file siteCache.idx.php. It contains all kinds of stuff prefetched from the database (system settings, snippets, chunks...) and among other things there are also three large arrays
$aliasListing,
$documentListing and
$documentMap (I call them
document arrays) that contain some pregenerated information about every document on the site. These arrays can be used in many different ways but one of their main purposes is to make generating document urls as fast as possible. Obviously the size of these arrays (and the siteCache.idx.php file) grows with the size of the website. Once it reaches certain point (according to the original FAQ for MODX Evo it is around 5000 documents), things might get weird.
What actually happens when the limit is reached
If you thought that the website would get unusably slow, then you would be wrong. The siteCache.idx.php is a standard PHP file which is included at the beginning of every request. This is actually pretty fast even with a lot of documents, but the problems is that Apache/PHP is not built to frequently include extremely large PHP files (3.5MB for 11500 documents). My problems started with around 4000 documents and the website was basically crashing the whole shared server every few hours. My webhoster found that out and shut our website down until we fixed it.
But that's not all. I applied few tweaks to get the website running and a few thousands of documents later, the memory started to be the issue. Simply the arrays were too big and the available memory wasn't enough. I had to ask the webhoster to increase the memory limit for our website (it's 512M now).
Anyway, one thing to remember is that the number 5000 is only approximate and it may vary depending on your server configuration.
How my current solution works
The basic idea behind my solution is to move the pregenerated content of the document arrays into a MySQL table. It basically means that instead of including 3.5MB file you have to fetch 11500 records from the database every request. It's naturally slower but it is still reasonably fast and what is more important, it works and it is reliable. Refreshing the content of the cache table is also fast enough (although slower than generating siteCache.idx.php). Naturally, the content of the cache table has to be converted to the actual document arrays in order to work with the rest of the MODX. My solution provides two different modes:
- In the classic mode (MODE_CLASSIC), the document arrays are generated completely and look just like they came from the siteCache.idx file in the original MODX. This mode is definitely more reliable (since is 100% compatible with the original MODX) but is slower and more memory consuming.
- In the fake mode (MODE_FAKE), special objects are used as the document arrays. They imitate the original arrays in a way that they look just like arrays (they implement interfaces ArrayAccess and Iterator) but they don't contain any actual data and work just as a link to the cache table from where the data is fetched only when required.
The "fake" mode
The classic mode is actually pretty simple and there is not much to talk about. It's a reliable solution that you can safely use on your website with minimal changes to the MODX core. The fake mode is, however, a little bit more interesting and before you try to use it, you should know about its pros&cons. First, how does it work.
- When the object is accessed using a key (i. e. $aliasListing[$id]), the required data is fetched quickly from the database using one simple SQL query.
- When the object is iterated over (i. e. foreach ($documentListing as $path => $id) {} ), one SQL query is made to fetch all records from the cache table and the records are then returned one by one.
This follows a simple philosophy that when rendering a page we most likely won't need the information about all documents. Unfortunately, the MODX core is written in a way that doesn't share this philosophy and is not very efficient with these fake arrays. The most problematic are snippets of code similar to this:
foreach ($documentListing as $path => $id) {
$a = $aliasListing[$id];
}
This would generate one SQL query for each document (in my case 11500 queries). Luckily, the fake arrays are smart and they share the database resource. So when we are iterating over $documentListing, $aliasListing has access to the current row from the cache table as well without explicitly querying it. Unfortunately, there are cases when even this approach fails. See the following code:
$tmp = array();
foreach ($documentListing as $path => $id) {
$tmp[$id] = $path;
}
foreach ($tmp as $id => $path) {
$a = $aliasListing[$id];
}
This is extremely inefficient and unfortunately there is no other way to get around this than actually tweak the core. The tweaks will be provided later.
Last thing to know about these fake arrays is that although they implement ArrayAccess and Iterator interfaces and therefore it is possible to use them as arrays, the imitation is not perfect. For example most PHP array function won't work on these objects. But I have tested it against the standard MODX installation (core + common extras like Ditto, Wayfinder etc) and it doesn't seem to be a problem. However, you might be using some snippet/plugin/module that won't work with these arrays so you should know about it. Anyway, you can find more information about the implementation directly in the code, it's heavily documented.
Installation
The following procedure and the attached code is tested on the current version of Modx Evo
1.0.12. There have been some noticeable changes in the latest releases so it probably won't work on older versions. At least PHP 5 is required (PHP 5.3 is recommended).
1. Download the attached archive with the source code and unzip it to
/manager/includes. A new folder called
modx_db_cache should appear.
2. Create the cache table in your database with the following schema:
CREATE TABLE `modx_cache` (
`id` int(10) NOT NULL,
`parent` int(10) NOT NULL,
`path` varchar(255) NOT NULL,
`alias` varchar(255) NOT NULL,
`fullpath` varchar(255) NOT NULL,
`isfolder` int(1) NOT NULL,
PRIMARY KEY (`id`),
KEY `fullpath` (`fullpath`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Make sure to change the name according to your table prefix (the base name of the table is just
cache).
3. Open the file
/manager/processors/cache_sync.class.processor.php and locate the part when the document arrays are generated. It should be lines
197 to 218. Delete/comment them and put the following code instead:
$tmpPHP .= '
include_once(MODX_BASE_PATH."manager/includes/modx_db_cache/ModxDbCache.php");
$dbCache = ModxDbCache::getInstance($this);
$dbCache->setMode(ModxDbCache::MODE_CLASSIC);
$this->aliasListing = $dbCache->getAliasListing();
$this->documentListing = $dbCache->getDocumentListing();
$this->documentMap = $dbCache->getDocumentMap();
';
include_once(MODX_BASE_PATH."manager/includes/modx_db_cache/ModxDbCache.php");
$dbCache = ModxDbCache::getInstance($modx);
$dbCache->update();
4. Clear the cache.
That's it. You're now using my alternative caching method in the
classic mode. As you can see, the changes done to the core were really minimal. If you want to try the fake mode, just replace
MODE_CLASSIC with
MODE_FAKE. But in that case I would highly recommend you to apply following two tweaks in order to get the best performance.
Tweaks for the "fake" mode.
Both tweaks apply to the file
/manager/includes/document.parser.class.inc.php.
1. Find the method
getChildIds and replace it with the following code:
function getChildIds($id, $depth= 10, $children= array ()) {
// Initialise a static array to index parents->children
static $documentMap_cache = array();
static $aliasListing_cache = array();
if (!count($documentMap_cache)) {
foreach ($this->documentMap as $document) {
foreach ($document as $p => $c) {
$documentMap_cache[$p][] = $c;
// By tobice: We create a cached version of reduced
// $aliasListing as well. Fake arrays $aliasListing and
// $documentMap share the same database resource therefore
// accessing the information about current document is
// fast and a new query is not required. We can use this
// prepared data in the second loop. We don't have to copy
// $aliasListing with all information since we won't need
// it. Just $pkey (path) and $c (id) is enough.
$pkey = (strlen($this->aliasListing[$c]['path']) ? "{$this->aliasListing[$c]['path']}/" : '') . $this->aliasListing[$c]['alias'];
$aliasListing_cache[$c] = $pkey;
}
}
}
// Get all the children for this parent node
if (isset($documentMap_cache[$id])) {
$depth--;
foreach ($documentMap_cache[$id] as $childId) {
// By tobice: Instead of accessing $aliasListing which would
// trigger an SQL query, we use the cached version of
// $aliasListing that we prepared before.
$pkey = $aliasListing_cache[$childId];
if (!strlen($pkey)) $pkey = "{$childId}";
$children[$pkey] = $childId;
if ($depth) {
$children += $this->getChildIds($childId, $depth);
}
}
}
return $children;
}
This tweak is essential for the fake mode. It's here to eliminate the problem with the two different cycles described somewhere above that generates one SQL query for each document in the database.
2. Find the method
rewriteUrls and replace it with the following code:
function rewriteUrls($documentSource) {
// by tobice: faster method to generate URLs.
// Notice that the regexp to find the links is slightly modified. It
// contains "/?" at the beginning to absorb a possible slash at the
// beginning. It will be automatically generated by makeUrl()
$in = '!/?\[\~([0-9]+)\~\]!ise'; // Use preg_replace with /e to make it evaluate PHP
$out = '$this->makeUrl("\\1")';
$documentSource = preg_replace($in, $out, $documentSource);
return $documentSource;
}
This tweak can make a huge difference, even when the cache runs in the classic mode. Usually, there are no more than 50 links on one page and therefore it is really unnecessary to loop over all documents. With the combination of the fake arrays, this can significantly speed up the rendering time of cached documents because information about 99% documents won't be fetched at all.
Performance comparison
I didn't run any exact tests but based on the rendering times measured by MODX I can give you a basic idea about the performance. So if you're wondering about the speed, the original MODX cache is still the fastest and I would recommend you to use it as long as possible. The performance of the fake mode highly depends on the system and the PHP version (the implementation of ArrayAccess and Iterator can be different over various versions of PHP). On my dev machine (very fast i7, SSD, lots of RAM, PHP 5.3.17), the fake mode is about the same speed as the classic mode for uncached documents, but is significantly faster for cached documents. On the production server (unknown hw spec, PHP 5.3.10), the fake mode is to my disappointment very slow for both uncached and cached documents. Still, one big advantage of the fake mode is that it consumes much less memory. So at some point, I may be forced to use the fake mode despite its slow speed.
[ed. note: tobice last edited this post 12 years, 11 months ago.]
☆ A M B ☆
