We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 32667
    • 38 Posts
    I was able to migrate my 0.9.6.1 site to Revolution no problem. I would gladly use Revolution now for my project but I need certain functionality (WebLoginPE, Jot) that don’t seem to be ready at this point, and I’ve only got 3 weeks to work with.

    So I think I am stuck with using 0.9.6.1 for this project.

    Details:

    • ~6000 documents. Most of the documents are directory listings.
    • Each document would have approx. 10 template variables.

    I’ve read about the ~5000 document limit and it seems some people have hacked the core to go around, but there isn’t information on how to do so.

    From what I’ve gathered, modx chokes when the siteCache.idx.php file gets too large. I edited the site cache processor so it doesn’t output the contentTypes (all my docs are text/html and this is the default for the type variable in the document parser). So that will save some size. I’ve moved my snippets so they are called from a single snippet, as per OpenGeek’s suggestion.

    I have about 400 docs currently and my siteCache is 160kb.

    What else can I do to make 6k docs feasible? Thanks for your time.

    edit: Mods - I see the stickied topic above. Sorry, I should have posted this there.
      • 28042 ☆ A M B ☆
      • 24,524 Posts
      I noticed that there is a list of document aliases for each document in the cache. I would imagine 6000 rows of aliases would enlarge the cache considerably. If you could see your way to using an alias prefix, like "page", and leaving the alias blank so it will use the doc ID instead (resulting in URLs like page35.html) I think it would cut down on the size a lot. If you’re using friendly alias paths, you could give your parent documents aliases so you would have the path as a proper alias; there shouldn’t be so many of those that it would load up the cache file.
        Studying MODX in the desert - http://sottwell.com
        Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
        Join the Slack Community - http://modx.org
        • 28042 ☆ A M B ☆
        • 24,524 Posts
        You can also move a lot of chunks that you are using as templates for Wayfinder or Ditto to external config files instead of using chunks.
          Studying MODX in the desert - http://sottwell.com
          Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
          Join the Slack Community - http://modx.org
          • 32667
          • 38 Posts
          Quote from: sottwell at Aug 15, 2008, 04:22 AM

          I noticed that there is a list of document aliases for each document in the cache. I would imagine 6000 rows of aliases would enlarge the cache considerably. If you could see your way to using an alias prefix, like "page", and leaving the alias blank so it will use the doc ID instead (resulting in URLs like page35.html) I think it would cut down on the size a lot. If you’re using friendly alias paths, you could give your parent documents aliases so you would have the path as a proper alias; there shouldn’t be so many of those that it would load up the cache file.

          Thank you for the suggestion Susan, but I need aliases to function so I started hacking away by removing all alias lines from the siteCache (uncomment all lines in cache_sync.class.processor.php involving the alias array -> ~line 112,113,130), knocking off 25% of the siteCache filesize and providing a slight speed improvement.

          Then I did this:
          The siteCache contains a line for every document’s alias, such as:
          $a[8] = array('id' => 8, 'alias' => 'search-results', 'path' => '');

          I don’t need the alias line for every doc because I can rewrite the url using the documentListing variable. Example:
          $d['search-results'] = 8;

          So in document.parser.class.inc.php in the rewriteUrls function, I removed the foreach that iterates thru $this->aliasListing and changed it to:
          foreach($this->documentListing as $k => $v)
          {
              $aliases[$v] = $k;
          }
          


          Since snippets like AjaxSearch just pass the docid to the makeUrl function and the makeUrl function uses $this->aliasListing (which we removed), I had to fix makeUrl function in doc parser file. I changed:
          elseif ($this->config['friendly_urls'] == 1 && $alias == '') {
                      $alias= $id;
                      if ($this->config['friendly_alias_urls'] == 1) {
                          $al= $this->aliasListing[$id];
                          $alPath= !empty ($al['path']) ? $al['path'] . '/' : '';
                          if ($al && $al['alias'])
                              $alias= $al['alias'];
                      }
                      $alias= $alPath . $this->config['friendly_url_prefix'] . $alias . $this->config['friendly_url_suffix'];
                      $url= $alias . $args;
          }

          to:
           elseif ($this->config['friendly_urls'] == 1 && $alias == '') {
                      if ($this->config['friendly_alias_urls'] == 1) {
                          $al = array_search($id,$this->documentListing);
                          $alias = ($al == TRUE) ? $al : '';
                      }
                      $alias= $alPath . $this->config['friendly_url_prefix'] . $alias . $this->config['friendly_url_suffix'];
                      $url= $alias . $args;
          } 


          This is similar thinking to what andytwiz said he did in the other similar thread about this subject, although he didn’t cache anything and retrieved necessary info from database.

          I have 405 documents now and my siteCache is 130kb. Will be knocking it down to about 40kb when I move all chunks and snippets to external files.
            • 36416
            • 589 Posts
            Quote from: woundedbeer at Aug 14, 2008, 07:32 PM

            From what I’ve gathered, modx chokes when the siteCache.idx.php file gets too large. I edited the site cache processor so it doesn’t output the contentTypes (all my docs are text/html and this is the default for the type variable in the document parser). So that will save some size.

            This is interesting idea - probably it would be wise to set default contentType to text/html (or in site config database), and then store only types that are different from default in the siteCache. Anyone with an idea where is content type cache referenced before I start grepping around?
              • 32667
              • 38 Posts
              Quote from: Eol at Jan 09, 2009, 09:15 AM

              Quote from: woundedbeer at Aug 14, 2008, 07:32 PM

              From what I’ve gathered, modx chokes when the siteCache.idx.php file gets too large. I edited the site cache processor so it doesn’t output the contentTypes (all my docs are text/html and this is the default for the type variable in the document parser). So that will save some size.

              This is interesting idea - probably it would be wise to set default contentType to text/html (or in site config database), and then store only types that are different from default in the siteCache. Anyone with an idea where is content type cache referenced before I start grepping around?

              This is exactly what you want to do.

              The default content type is text/html. There is no need to write this to the site cache because the document parser has text/html set as the default if the value is empty.

              So in /manager/processors/cache_sync.class.processor.php, to get rid of content type lines (and only write non text/html) in site cache file, edit this line by adding a where clause

              Change:
              $sql = "SELECT id, contentType FROM $dbase.`".$table_prefix."site_content`"; 
              


              To:
              $sql = "SELECT id, contentType FROM $dbase.`".$table_prefix."site_content` WHERE contentType != 'text/html'"; 
              


              And voila, it will only write the contentType line to the site cache file if it is different from text/html. Out of the 5k docs I have, I only have a few xmls.

              That is all you need to do!

              I also removed the contentType from the Select in the //get aliases modx: support for alias path section, at/around line 110

              Just change:
              $sql = "SELECT IF(alias='', id, alias) AS alias, id, contentType, parent FROM $dbase.`".$table_prefix."site_content` ORDER BY parent, menuindex";
              


              to:
              $sql = "SELECT IF(alias='', id, alias) AS alias, id, parent FROM $dbase.`".$table_prefix."site_content` ORDER BY parent, menuindex";
              
                • 4749
                • 623 Posts
                Woundedbeer, have you tried this? If so, how is it working for you? Have you noticed any adverse effects?
                  The MODx has you...
                  Utah Web Design
                  • 32667
                  • 38 Posts
                  Yes, I have done all of these modifications and there are no adverse effects. Been using these hacks for months and it improves performance! I’ve been able to write a significantly smaller siteCache.idx.php file. Once you start getting up over 5k docs, this file would normally bloat so much that modx starts to choke.

                  Here are my stats:

                  • 4,385 documents
                  • siteCache file size = 527kb
                  • If I move the chunks and snippets to external files, then the siteCache file would be 411kb

                  You can look at your siteCache file and see everything that it writes.

                  Mine only writes:
                  the config array (about 80 lines)

                  Then I have the documentListing and documentMap lines for each doc so an example is:

                  $d[’contact’] = 77;
                  $m[] = array(’0’ => ’77’);

                  Then, with the last mod, I only write the contentType array for the docs that aren’t text/html (only 1 line for me).

                  I have attached the modified cache_sync.class.processor.php and document.parser.class.inc.php files in the zip file below. This is what I am running on 0.9.6.1p2.
                    • 4749
                    • 623 Posts
                    Thanks. I’ll take a closer look today. I’ve got a site with about 2000 docs and my sitecache file is already 570k. Snippets, modules, and chunks as includes will take out over 8K lines in the sitecache file which is about 14k right now. So, this sounds very interesting to me.

                      The MODx has you...
                      Utah Web Design
                      • 32667
                      • 38 Posts
                      Quote from: cbaone at Jan 10, 2009, 12:13 PM

                      Thanks. I’ll take a closer look today. I’ve got a site with about 2000 docs and my sitecache file is already 570k. Snippets, modules, and chunks as includes will take out over 8K lines in the sitecache file which is about 14k right now. So, this sounds very interesting to me.



                      No problem.

                      I would definitely move your snippets and such to external files. I created one snippet called Include which contains the following (courtesy of Jason/OpenGeek):
                      <?php
                      // &file=`assets/foo/bar.php` indicates the file to include
                      // &return=`1` return explicit content from return statement in included file
                      //  default behavior is to use output buffering to capture the content from the include
                      
                      $output = '';
                      
                      if (isset($file)) {
                      $file = "/public_html/assets/snippets/" . $file;
                          if (!isset($return) || $return == false) {
                              ob_start();
                              require($file);
                              $output = ob_get_contents();
                              ob_end_clean();
                          } else {
                              $output = include ($file);
                          }
                      }
                      return $output;
                      ?>


                      Then when I need to refer to a snippet, instead of [!SnippetName!], I do something like this:

                      [!Include? &file=`webloginpe/webloginpe.snippet.php` &return=`1` &type=`register` &Type=`instant`!]

                      Hope this helps.