We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 9207 ☆ A M B ☆
    • 2,475 Posts
    The short answer is that I frequently want to load "chunks" from files or from a web service, or from my own database tables. It’s such a common task to want to just *parse* out the placeholders and not necessarily make a trip to the database. I don’t like the getChunk function because it does 2 things at once, when I only want one of those things. E.g. if I’m creating a whole series of small templates to use while generating a form, it gets to be extremely inefficient to create a chunk in the database for each one. It’s much more efficient to create a config file that lists 100s of small tpls in one place, but it’s hard to make MODx use those without reinventing the wheel.

    The $x->process method offers some relief there for modChunk objects, but I’m still hoping for direct access to the MODx parser.

    Jason, we’ve talked about this before: say I’ve got a custom resource that gets its values from a custom database table and not via MODx TVs. Last we spoke, it was not possible to pass that object through a template object for formatting. Instead the template was always pulled from the original resource, making it necessary to do manual parsing. You know what I mean? Sorry, some of the details here are fuzzy.

    In my mind, I’m doing this all the time (be it with chunks or templates or resources):
    1. Load up a resource with attributes (e.g. a "Book" resource).
    2. Load up something that defines a format (i.e. a view -- a chunk or template with placeholders for [[+title]] and [[+author]]).
    3. Parse it, return resulting HTML to requesting page.

    I crawled through a lot of the class functions when I did a site heavy with custom database tables, but I wasn’t able to accomplish it in what I felt was a clean way.
      • 22303 MODX Staff
      • 10,725 Posts
      Quote from: Everett at Nov 08, 2010, 04:27 PM

      The short answer is that I frequently want to load "chunks" from files or from a web service, or from my own database tables. It’s such a common task to want to just *parse* out the placeholders and not necessarily make a trip to the database. I don’t like the getChunk function because it does 2 things at once, when I only want one of those things. E.g. if I’m creating a whole series of small templates to use while generating a form, it gets to be extremely inefficient to create a chunk in the database for each one. It’s much more efficient to create a config file that lists 100s of small tpls in one place, but it’s hard to make MODx use those without reinventing the wheel.

      The $x->process method offers some relief there for modChunk objects, but I’m still hoping for direct access to the MODx parser.
      The process() method is available for every modElement derivative class, and every modTag derivative class. It is what ensures consistency in the parsing steps. That said, you can use the modParser::parseElementTags() method directly, just be careful what comes out of the box you open. wink

      As for inefficiency on Chunks, they should all be cached in the $modx object during the request, in the Resources they appear in, and potentially in the result-set cache when enabled. I think scalability and efficiency are very high on the priority list here. In fact, if you look at the file-based Chunk implementations that splittingred uses in most of his Extras, as well as the one I use in getResources for allowing file-based Chunks, it’s as simple as creating a modChunk in memory (no database activity required), and calling process() with both parameters, the second one being the content you are getting from your preferred source.
      <?php
      $chunk = $modx->newObject('modChunk', array('name' => $key));
      $chunk->setCacheable(false);
      $output = $chunk->process($properties, $content);
      ?>


      Quote from: Everett at Nov 08, 2010, 04:27 PM

      Jason, we’ve talked about this before: say I’ve got a custom resource that gets its values from a custom database table and not via MODx TVs. Last we spoke, it was not possible to pass that object through a template object for formatting. Instead the template was always pulled from the original resource, making it necessary to do manual parsing. You know what I mean? Sorry, some of the details here are fuzzy.
      The example of passing your data through a "template object", aka Chunk, is above. Sorry, maybe I’m a little fuzzy... laugh

      Quote from: Everett at Nov 08, 2010, 04:27 PM

      In my mind, I’m doing this all the time (be it with chunks or templates or resources):
      1. Load up a resource with attributes (e.g. a "Book" resource).
      2. Load up something that defines a format (i.e. a view -- a chunk or template with placeholders for [[+title]] and [[+author]]).
      3. Parse it, return resulting HTML to requesting page.

      I crawled through a lot of the class functions when I did a site heavy with custom database tables, but I wasn’t able to accomplish it in what I felt was a clean way.
      You write a Snippet that looks up the requested book data, you set placeholders, and maybe return some content from the snippet, which then get’s parsed. I still don’t see the need to manually parse Elements of content in MODx. And iterating with Chunks, be they from the database, or transient in your Snippet as illustrated above, is pretty straightforward with the process() method, no?
        • 9207 ☆ A M B ☆
        • 2,475 Posts
        Thanks, Jason. I’ll have to dust off the code I was working on ~6months ago where I was trying to use MODx templates to format stuff from custom tables.
          • 25037
          • 32 Posts
          Quote from: OpenGeek at Nov 08, 2010, 04:19 PM

          If you look in the process() method of any modElement (or modTag) derivative class, you will see the part that applies to a single element. But the majority of the parsing process starts with the Resource and ends in modResponse::outputContent().
          I hope my question isn’t too off-topic. Does anyone have the time to explain why the parser processElementTags() method is called twice in modResponse::outputContent()? The difference in the following 2 calls is the fourth parameter named $removeUnprocessed:
          $this->modx->parser->processElementTags('', $this->modx->resource->_output, true, false, '[[', ']]', array(), $maxIterations);
          $this->modx->parser->processElementTags('', $this->modx->resource->_output, true, true, '[[', ']]', array(), $maxIterations);
          

          Is the second call simply to remove all tags that could not be processed for whatever reason in the first parsing?
            • 22303 MODX Staff
            • 10,725 Posts
            Quote from: rickj at Nov 09, 2010, 04:23 PM

            I hope my question isn’t too off-topic. Does anyone have the time to explain why the parser processElementTags() method is called twice in modResponse::outputContent()? The difference in the following 2 calls is the fourth parameter named $removeUnprocessed:
            $this->modx->parser->processElementTags('', $this->modx->resource->_output, true, false, '[[', ']]', array(), $maxIterations);
            $this->modx->parser->processElementTags('', $this->modx->resource->_output, true, true, '[[', ']]', array(), $maxIterations);
            

            Is the second call simply to remove all tags that could not be processed for whatever reason in the first parsing?
            That’s correct; a set of passes are done to process non-cacheable tags up to 10 full recursive iterations or until no tags are left in the output, leaving any that do not get processed in place. The final pass then starts removing the unprocessed tags as it encounters them.
              • 25037
              • 32 Posts
              Interesting. If the second call to processElementTags only removes tags, then there may be a better way. I’ve been looking at recursive regular expressions, and this regex will remove modx style tags including nested tags:
              /\[\[(?>[^\[\[\]\]]]+|(?R))*\]\]/

              The escaping backslashes make it messy, but it’s really terse. So the second call to processElementTags() could change to this:
              $prefix = preg_quote('[[');
              $suffix = preg_quote(']]');
              $regex = '/' . $prefix . '(?>[^' . $prefix . $suffix . ']+|(?R))*' . $suffix . '/';
              if($result = preg_replace($regex ,'',$this->modx->resource->_output))
              {
                 $this->modx->resource->_output = $result;
              }
              

              I have only tested this change with a simple template, but it looks good.

              Further to this, I have changed the modParser::collectElementTags() method to use the same regex. It probably improves performance and uses 4X less lines of code:
                       public function collectElementTags($origContent, array &$matches, $prefix= '[[', $suffix= ']]') {
              		$regex = '/' . preg_quote($prefix) . '(?>[^' . preg_quote($prefix) . preg_quote($suffix) . ']+|(?R))*' . preg_quote($suffix) . '/';
              		$results = array();
              
              		if (!empty ($origContent) && is_string($origContent) && strpos($origContent, $prefix) !== false) {
              
              			// All tags found will be stored in an array within the first index of $results (within $results[0])
              			if(preg_match_all($regex,$origContent,$results)) {
              				/*
              				An example of what the $results array will now look like:
              				Array
              				(
              					[0] => Array
              						(
              							[0] => [[$my_chunck? &replace=`[[$chunk_to_replace]]`]]
              							[1] => [[$my_snippet]]
              						)
              				)
              
              				Below, it is converted to to match the output of the current collectElementTags() method:
              
              				Array
              				(
              					[0] => Array
              						(
              							[0] => [[$my_chunck? &replace=`[[$chunk_to_replace]]`]]
              							[1] => $my_chunck? &replace=`[[$chunk_to_replace]]`
              						)
              
              					[1] => Array
              						(
              							[0] => [[$my_snippet]]
              							[1] => $my_snippet
              						)
              				)
              				*/
              				$prefixLength = strlen($prefix);
              				$suffixLength = strlen($suffix);
              				$innerTagLength = $prefixLength + $suffixLength;
              				foreach($results[0] as $result) {
              					$matches[] = array($result, substr($result, $prefixLength, strlen($result) - $innerTagLength));
              				}
              			}
              		}
              
              		return sizeof($results[0]);
              	}
              



                • 28215
                • 4,149 Posts
                rick, can you file that as an improvement here: http://bugs.modx.com/ so it doesnt get lost on the forums?
                  shaun mccormick | bigcommerce mgr of software engineering, former modx co-architect | github | splittingred.com
                  • 25037
                  • 32 Posts
                  Quote from: splittingred at Nov 10, 2010, 02:34 PM

                  rick, can you file that as an improvement here: http://bugs.modx.com/ so it doesnt get lost on the forums?
                  Ok
                  http://bugs.modx.com/issues/2763
                    • 5811
                    • 1,717 Posts
                    I have done a short test with the optimisation proposed by nick.

                    I parse a simple document (#29) which contains: [[!getResources? &parents=`3` &limit=`100` &tpl=`blogPost`]]
                    with blogPost:
                    <div class="blogPost">  
                      <h2><a href="[[~[[+id]]]]" title="[[+pagetitle]]">[[+pagetitle]]</a></h2>  
                       
                      <p class="author"><strong>Author:</strong> <span class="author">[[+createdby:userinfo=`username`]]</span></p>  
                      <p class="date">[[+publishedon:strtotime:date=`%b %d %Y`]]</p>  
                      <p class="summary">[[+introtext]]</p>  
                      <p class="readmore"><a href="[[~[[+id]]]]"><span>Read more</span></a></p>  
                       
                      <div class="clear"></div>  
                    </div>


                    -> limit = 5 (default)
                    with the Nick’ optimisation, the elapsed time is of 0.3791 s
                    with the current solution : Elapsed time : 0.3782 s

                    -> with &limit=`100` and 18 documents displayed
                    with the optimisation the elapsed time is of 1.1179 s
                    with the current solution : Elapsed time : 1.0682 s

                    Before each time the MODx & firefox caches have been cleared.

                    May be some tests with more complicated tags (several embedded tags) should be tested, but I am not sure that the gain of optimisation is real.
                    May be the recursive loop is faster than the use of a regexp.

                    I did the tests with the following code (document #29 parsed)
                    <?php 
                    
                    define('MODX_BASE_URL', 'http://localhost/www/modx204/');
                    
                    define('MODX_BASE_PATH', 'E:/www/modx204/');
                    require_once MODX_BASE_PATH.'config.core.php';
                    
                    require_once MODX_CORE_PATH.'config/'.MODX_CONFIG_KEY.'.inc.php';
                    require_once MODX_CORE_PATH.'model/modx/modx.class.php';
                    
                    $modx = new modX();
                    $modx->initialize('web');
                    define('MODX_API_MODE', true);
                    
                    // get not deleted, published and searchable documents
                    $documents = $modx->getCollection('modResource',array(  
                        'deleted' => 0,
                        'published' => 1,
                        'searchable' => 1,
                        'context_key' => 'web',
                        'id' => 29
                    )); 
                    
                    if (!empty($documents)) { 
                    
                        echo '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
                    <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
                    <base href="http://localhost/modx204/" />
                    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
                    <head>
                    </head>
                    <body>'."\n\r";
                    
                        $mtime = microtime();
                        $mtime = explode(" ", $mtime);
                        $tstart0 = $mtime[1] + $mtime[0];
                            
                        $i = 1;
                        foreach ($documents as $document) {
                            echo "id: " . $document->get('id')." index: " . $i ."<br />\n\r";
                            // process content
                            // get unparsed content
                            $content = $document->get('content');
                            
                            if (!empty($content)) { 
                            
                                $modx->resource =& $document;
                                
                                // get the max iterations tags are processed before processing is terminated 
                                $maxIterations= (integer) $modx->getOption('parser_max_iterations', null, 10);
                    
                                $mtime = microtime();
                                $mtime = explode(" ", $mtime);
                                $tstart = $mtime[1] + $mtime[0];
                            
                                // parse all cacheable tags first 
                                $modx->getParser()->processElementTags('', $content, true, false, '[[', ']]', array(), $maxIterations);
                    
                                // parse all non-cacheable and remove unprocessed tags 
                                $modx->getParser()->processElementTags('', $content, true, true, '[[', ']]', array(), $maxIterations);    
                              
                                echo $content;
                                
                                $mtime = microtime();
                                $mtime = explode(" ", $mtime);
                                $tend = $mtime[1] + $mtime[0];
                    
                                $eTime= ($tend - $tstart);
                                $eTime= sprintf("%2.4f s", $eTime);
                                echo "Elapsed time : ". $eTime . "<br />\r\n";
                            }      
                        $i++;
                        }
                        $mtime = microtime();
                        $mtime = explode(" ", $mtime);
                        $tend = $mtime[1] + $mtime[0];
                    
                        $eTime= ($tend - $tstart0);
                        $eTime= sprintf("%2.4f s", $eTime);
                        echo "Total elapsed time: ". $eTime."<br />\n\r";
                    
                        echo '</body>
                    </html>'."\n\r";
                    
                    }
                    ?>
                      • 25037
                      • 32 Posts
                      Thanks coroico. I’m happy the changes worked for you at least.

                      The only crude benchmark that I did was to extract the existing::collectElementTags() and compare that with my version using a simple string with tags within a loop. I was getting anywhere between a 3X to 6X speed improvement depending on the length of string and number of tags I was using. It seems that within the scope of a revo application, my changes would seem pretty insignificant for performance.

                      I think my changes have merit when considering code simplicity, if not performance.