We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 9849
    • 74 Posts
    I read through the whole discussion and unfortunately cannot comment other than that I am disappointed. I did some small websites with MODx and never had to even think about limitations for documents. But know I am facing a project where I have to move about 12.000 documents (news articles way back until 2003, this makes 2.400 documents each year). It seems to be impossible to archive with MODx at the moment, right?
    Quote from: sottwell at May 18, 2008, 10:06 AM

    … 0.9.7 is coming along …
    But does that change anything? I mean does it lift the documents limit? I am a pragmatical web designer and need fact (as my customers need them even more). I understand that no one will ever be able to tell me: “You can use 10.583 documents exactly”. But I would really appreciate an approximate number of documents which 0.9.7 (or 2.0 revolution if you want to) can deal with.

    Thanks in advance.
      • 33372
      • 1,611 Posts
      0.9.7 is a completely different animal and it’s caching system will not have the specific bottleneck that limits you now. So effectively this issue will not exist in MODx 0.9.7 as far as I know. Other factors may impose practical limits to the number of documents that you can work with, but I expect that the number we’d be talking about would be exponentially higher than what it is now.
        "Things are not what they appear to be; nor are they otherwise." - Buddha

        "Well, gee, Buddha - that wasn't very helpful..." - ZAP

        Useful MODx links: documentation | wiki | forum guidelines | bugs & requests | info you should include with your post | commercial support options
        • 17282
        • 283 Posts
        Quote from: OpenGeek at Oct 10, 2007, 09:16 AM

        Quote from: MasterzDee at Oct 10, 2007, 08:49 AM

        Sooner or later MODx must support large docs tree and dirs (30.000 plus). This days 5000 docs is nothing.
        Because everyone wants a site with 30,000 pages? I’ve never even come close to building one with a hundred pages, at least where more than a dozen or so were actually ever worth reading. But seriously, IMHO, anything with that large a quantity of articles needs special attention, regardless of what product you are using.

        In any case, MODx will support what it can based on a number of factors, mostly related to the limitations of PHP and the software/hardware environment it is running within, as well as the architecture. Several changes are coming in the new releases that should help you better organize documents into distinct sections that can live by themselves, but large numbers of articles in the same sections are still going to be an issue. This is why some content needs to be managed in ways other than the standard MODx web content management paradigm (i.e. documents in the tree).

        Finally, feel free to make suggestions and feature requests for MODx all you want, but keep in mind just telling us that MODx must do something is likely to accomplish nothing.


        largest ive built so far is an ongoing new reviews project standing at 1800 pages ...
          Everytime you use Flash ... a puppy dies .....
          R.G Taylor
          • 3749
          • 24,544 Posts
          Quote from: OpenGeek at Jun 16, 2008, 09:50 AM

          Create one snippet called ’include’, then simply put your snippets in external files and include them ...

          Do you do this in MODx Revolution also?
            Did I help you? Buy me a beer
            Get my Book: MODX:The Official Guide
            MODX info for everyone: http://bobsguides.com/modx.html
            My MODX Extras
            Bob's Guides is now hosted at A2 MODX Hosting
            • 22303 MODX Staff
            • 10,725 Posts
            Quote from: BobRay at Sep 26, 2008, 01:05 AM

            Quote from: OpenGeek at Jun 16, 2008, 09:50 AM

            Create one snippet called ’include’, then simply put your snippets in external files and include them ...
            Do you do this in MODx Revolution also?
            You can, though the benefit in Revolution is less, because all snippets are converted to global functions that are include_once()’d themselves (vs. being eval()’d).  That means two includes would be involved when using the include snippet (the Element loaded as an include + the files referenced in the include snippet itself), but this should not have a significant affect on performance overall.  The problem in the legacy releases is that all of the snippet source is stored in the siteCache.inc.php file, so the less source code in the snippets, the better.  This is not the case in Revolution.

            Just for further clarity, the cache files in Revolution are divided into context specific files and only include serialized versions of the xPDO objects representing plugins (which does include the source).  This is because plugins are executed much more often within their own contexts, while snippets are only executed where called; including snippets on demand makes more sense from a performance perspective, rather than burdening every request with all of the snippet source, which may or may not be eval()’d on a particular page.

            The next minor release of Revolution after 2.0 will include the ability to store the "source" of any Element on the filesystem without having to have a generic include snippet to mediate.  This will give developers who prefer to work with the filesystem a viable option for maintaining any MODx content they want, while still providing the database backed metadata necessary to drive the more important features of the MODx core engine.
              • 3749
              • 24,544 Posts
              Quote from: OpenGeek at Sep 26, 2008, 01:44 AM

              The next minor release of Revolution after 2.0 will include the ability to store the "source" of any Element on the filesystem without having to have a generic include snippet to mediate. This will give developers who prefer to work with the filesystem a viable option for maintaining any MODx content they want, while still providing the database backed metadata necessary to drive the more important features of the MODx core engine.

              I can’t wait. tongue
                Did I help you? Buy me a beer
                Get my Book: MODX:The Official Guide
                MODX info for everyone: http://bobsguides.com/modx.html
                My MODX Extras
                Bob's Guides is now hosted at A2 MODX Hosting
                • 4749
                • 623 Posts
                Quote from: OpenGeek at Jun 16, 2008, 09:50 AM

                Create one snippet called ’include’, then simply put your snippets in external files and include them, replacing the snippet calls with a modified tag that adds an additional parameter identifying the file containing the snippet code. Here is the include snippet I use on one site. It is the only snippet saved in my database, and prevents the source code for all those snippets from being included in the siteCache file. Parameters for other snippets will passthrough, unless there is a conflict on the parameter names I chose for the include snippet itself, but you can alter those to something unique. Note this also allows snippets that directly return content or those that echo content from the include, based on the optional &return parameter value.

                <?php
                // &file=`assets/foo/bar.php` indicates the file to include
                // &return=`1` return explicit content from return statement in included file
                //  default behavior is to use output buffering to capture the content from the include
                
                $output = '';
                if (isset($file) && file_exists($file)) {
                    if (!isset($return) || $return == false) {
                        ob_start();
                        include ($modx->config['base_path'] . $file);
                        $output = ob_get_contents();
                        ob_end_clean();
                    } else {
                        $output = include ($modx->config['base_path'] . $file);
                    }
                }
                return $output;
                ?>


                Jason, I may be a little confused on your advice, but how do you pass on all the individual parameters? Thanks for the code sample, but could you give an example of the call to replace a Ditto or Wayfinder call? Thanks.
                  The MODx has you...
                  Utah Web Design
                  • 22303 MODX Staff
                  • 10,725 Posts
                  Just change the snippet call to ’include’ and add the file and return parameters; all the other parameters for the snippet call would be the same as if you were calling the snippet the old fashioned way:
                  [[include? &file=`assets/snippets/wayfinder/wayfinder.snippet.php` &return=`1` &startId=`0` &outerTpl=`MyWayfinderOuterTpl`]]
                    • 4749
                    • 623 Posts
                    Thanks, Jason. I appreciate your time. It’s just as I thought. I guess I was over thinking it. Thanks. smiley
                      The MODx has you...
                      Utah Web Design
                      • 22303 MODX Staff
                      • 10,725 Posts
                      Quote from: ganeshXL at Dec 13, 2008, 08:00 AM

                      The devil is often in the details: e.g. if you write a custom snippet and you’re creating links: You loop through a mySQL result-set.... You could very well use $modx->makeUrl() to get the URL. However, if you know that you’re using SEO-friendly URLs, and you know how they’re constructed (alias with or without suffix etc.), you could omit this API call and simply concatenate pagetitle + suffix vars. Because I’m sure this API function is internally using again mySQL at least once.
                      Well, actually that function does not use the database at all. In fact, in order to avoid the MySQL calls when using makeUrl(), MODx caches all of the document entries along with needed information to construct the URLs in the siteCache.idx.php. And this brings us back to the original problem, which is that loading 10000 documents via siteCache on every request, in order to more quickly be able to generate URL’s dynamically at runtime, becomes less efficient than calling MySQL on demand, as you typically would not be creating URLs to all 10000 documents on every page.

                      Personally, I always run Ditto cached myself, opting to only have it regenerate when I edit the site. A caching strategy has to be considered when developing dynamic sites with 1000’s of documents; there is no way around it. This is why there is even more granularity and flexibility in managing caching options in Revolution, so custom caching strategies can be easily implemented and modified as a site grows.