[Usage/Code] Ditto performance & MySQL LIMIT

Hey everyone!

Yesterday I’ve experimented using MODx on a testground site with a very large quantity of resources (ex-docs), that is used as some catalog items, processed by Ditto into paged catalog.

As I don’t have any actual content, I’ve filled MODx with random-generated resources 1-4KB each, totally above 6000 pages for now.

The problems with ditto (showing nicely 10 items per page) started at about 4600 pages - somehow modx reaches the memory limit for PHP on my shared hosting account (32MB).

I’ve started looking through the ditto parameters, hoping to find some clues. I found one: it’s name is &queryLimit.

Unfortunately it works nothing like what I desired. As far as I understood from looking through the ditto source code, the snippet fetches from db everything that meets the criteria, even if it is to display only 10 items on the page. This means that for every of 600 pages in my catalog ditto fetches 6000 rows from mysql and this causes memory lacks. That’s very dissapointing.

&queryLimit=`10` works not like the MySQL LIMIT. In fact it just makes the db to return only 10 rows from result starting with row 0. So when I go to the page "2", db returns the same 10 rows, which were for the page "1", then ditto skips 10 items (as the start item is 10, set by $_GET[’start’]), and so ditto has 0 items to show, which it admits by showing "Resource Array Error" for each resource.

I understand that with all its filtering, sorting, etc. functionality ditto depends very much on PHP working with complete data fetched from database. That works fine but its really SLOW and what’s more its memory usage is linear dependendent on the total processed resources volume.

The best solution is to rewrite code so that the most of the functionality will be moved into MySQL (it has very optimized code, several layers of query optimization: this will execute much faster than on php and will leave more resources for the php).

For now I think to write my own snippet that will fit my today’s needs or to fix ditto - but that will take much more of my time.

Any suggestions/comments?

Regards,
xEviL

Hi,

it is not Ditto which takes 6000 rows and exhausts your 32 MB memory limit. This is a MODx’s caching system -> /assets/cache/siteCache.idx.php file. This file comprises map of all your 6000 dummy documents and is updated each time you are adding/removing document. MODx 2.0 where the caching system was rewritten from the scratch reportedly overcomes this 5000-6000 docs limit.

alik,
it is ditto. Cache works fine.

Page with error: http://tools.taurobots.com/ditto_6k.html

After removing the snippet call or limiting its query back to 4800 docs MODx work fine. Even with the cache enabled:

Here I set the limit to 100:

http://tools.taurobots.com/ditto_6k_limit100.html

works fine, all of 6000 docs are there:

http://tools.taurobots.com/wayfinder_menu.html

Regards,
xEviL

Try creating your own snippet that does what you want