We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 25663 MODX Staff
    • 12,272 Posts
    Quote from: R.S. at Nov 15, 2005, 10:06 AM

    Original parser:
    ~ 6.0 requests/second

    Modified parser:
    ~ 7.0 requests/second

    Looks good, no?

    Looks very good indeed... roughly a 17% improvement without touching the other parts that can also be cleaned up. smiley

    http://vt.shuis.tudelft.nl/~remon/test/
    Well, the bug/fun part begins! wink the ListIndexer call needs the format changed and the [*logout*] TV needs to be tweaked in the main navigation menu. Any chance you could also put up a second link to th original parser so we can see how that one performs in the same environment?

    Keep it up... looking great indeed. laugh
      Ryan Thrash, MODX Co-Founder
      Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
      • 25663 MODX Staff
      • 12,272 Posts
      And one follow-on question as well...

      Am I also correct in thinking we’ll pick up further speed by merging some of the database tables as well (e.g., chunks/snippets), and using the same parser calls to handle the various types of content (ditto)?

      If so... laugh laugh laugh!
        Ryan Thrash, MODX Co-Founder
        Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
        • 24253
        • 125 Posts
        Quote from: rthrash at Nov 15, 2005, 03:44 PM

        And one follow-on question as well...

        Am I also correct in thinking we’ll pick up further speed by merging some of the database tables as well (e.g., chunks/snippets), and using the same parser calls to handle the various types of content (ditto)?

        Sorry, but I don’t understand what you ask here, I’ll have to reread, but first a shower smiley

        And of course, I had a default install there allready, so have a look at the time difference:

        http://vt.shuis.tudelft.nl/~remon/modxdefault/

        Greetz,

        Remon
          • 25663 MODX Staff
          • 12,272 Posts
          From comparing the output the original modx parser (http://vt.shuis.tudelft.nl/~remon/modxdefault/) to the new recursive parser (http://vt.shuis.tudelft.nl/~remon/test/), it looks like we not only gain a LOT of functionality via nested calls, but we also pick up a modest performance boost as well!

          Original PHP Time: 0.1435
          After rev PHP Time: 0.0966

          So that roughly equates to a 45% speed boost... not bad eh?

          This brings up the point of caching. Remon seems to indicate that the calls via [! no longer work in the new parser, which means we should probably take a hard look at how we’re handling caching and try to figure out some way to do it automatically.

          The "rules" for caching never seemed to make sense to me anyway. For example, it seems that the menus always seem to work, whether cached or not. However, forms that post back to the page never seem to work unless not cached. Is it that we need another format to indicate non-caching?

          Remon, can you comment further?
            Ryan Thrash, MODX Co-Founder
            Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
            • 24253
            • 125 Posts
            Sure!

            Ehm, the number indeed seem to be correct, always difficult to have the exact same conditions!

            The parser indeed performs much better, keep in mind that a lot of stuf is behind our hands, so the total processing time gain for a page is not 45% !

            While working on the parser I thought a bit on the caching thing.

            It’s obvious that a "true" cached page (no merging of snippet that needs still to be processed) is the fastest way of serving a html page.

            It seems the [! !] call convention was made to have semi-cached pages. The content was cached, but the snippets were still executed.

            However, this is very confusing to use, and error prone IMHO.
            The user has to be aware that both calling a snippet like [[ ]] or [! !] has different effects, but also, when creating a page, he/she has to think about "do I want this page to be cached or not", and what effect does it have with the current template and the way the snippets are called there etc ?

            So, the fastest way to serve a page is to take it from cache, and the easiest way for the user is to not bother about caching at all!
            So, one calling convention for snippets/chuncks/etc and (perhaps) a global configuration parameter "enable/disable cache". (I’m thinking for those people having a NUMA system with 16 processors and 128 Gig of RAM but are going out of hard disk space hehe :-P )

            With one calling convention, the user doesn’t give the parser a hint if there can be done partial caching or not. I actually think, with one way to call snippets/etc there is no way to determine if a snippet output has to be regenerated or can be taken from cache.

            Why?
            Well, a snippet can depend on _any_ data inside the database, so there’s no way to say "this part in the database has changed, so we need to regenerate snippet X’s output".

            This however could be done perhaps for chunks, and others, but not for snippets.

            So, I suggest that by default, a page output is cached with a timestamp when it was created.
            Now, if the database changes, that time is written to a file, and when we check for a cached page, we compare these two values.
            If the database is newer, we generate a new page, and overwrite the old page cache for that particular page......

            This means that at least for snippets there is no more partial caching, perhaps chuncks can be cached (seperataly) easier....

            What do you think about this?

            Remon

            P.S.
            [! !] worked by replacing those with [[ ]] at the last stage and running the parser again on the source. We could still use this, but I don’t like it, and it doesn’t confirm to the calling standard

            P.S.2
            Just popping up into my mind.
            What if we "pregenerate" the page on document creation/modification, and cache it _without_ the snippets/TV’s processed?
            But then, why not generating the whole page at once?
            End then, why do we have a parser at all? ;-)
            I need to go to bed, since I obviously miss something here....
              • 25663 MODX Staff
              • 12,272 Posts
              The "right thing to do" would be to automatically handle the caching for the users and developers... a intelligent caching system. Now, how this is implemented is another question for which I don’t have an answer. To me it seems the common thread for disabling caching is for forms that need require a postback to the page or some other sort of similar function. How will this change with XML-RPC or AJAX calls, or will it at all for that matter!?!

              Having it be "automagical" would be ideal, but can it be done effectively?

                Ryan Thrash, MODX Co-Founder
                Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
                • 25663 MODX Staff
                • 12,272 Posts
                  Ryan Thrash, MODX Co-Founder
                  Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
                  • 32963
                  • 1,732 Posts
                  The solution IMO is that we first check to see if page caching is turned on:

                  Pseudo login

                  if ($modx->documentObject[’cacheable’]!=1) then
                     // page not cacheable therefore convert [! to [[
                     $modx->documentContent= str_replace(’[!’,’[[’,$modx->documentContent);
                  end if
                  // Parse document source
                  $this->documentContent = $this->parseDocumentSource($this->documentContent);

                  If the page is not "cacheable" we can then convert [! to [[ because nothing will be cached. But if the page is "cacheable" then we must first process [[ then [! just before page is sent to browser. See the $modx->outputContent() function:

                  // check for non-cached snippet output
                  if(strpos($this->documentOutput, '[!')>-1) {
                  	$this->documentOutput = str_replace('[!', '[[', $this->documentOutput);
                  	$this->documentOutput = str_replace('!]', ']]', $this->documentOutput);
                  
                  	// Parse document source
                  	$this->documentOutput = $this->parseDocumentSource($this->documentOutput);
                  }
                  


                  I would also recommend that we use the format [! ]] to indicate non-cacheable objects:

                  [!snippet]]
                  [!*content]]
                  [!+placeholder]]
                  [!{chunk]]

                  or we could use something like (I think Ryan had suggested this idea sometime ago):

                  [[!snippet]]
                  [[!*content]]

                  With this method we only have to parse the code once. When the system sees the "!" character it should store the tag inside a non-cached array:

                  $modx->nonCacheTags[] =  substr($tag,1);


                  This array could then be processed inside the $modx->outputContent() function. The $modx->nonCacheTags array could then be stored as:
                  $modx->documentObject['__NonCacheTags__'] = $modx->nonCacheTags

                  This way the system can retrieve the cache page and restore the $modx->nonCacheTags array to be pocessed inside the $modx->outputContent() function with the need to call parseDocumentsource().

                  What do you think?
                    xWisdom
                    www.xwisdomhtml.com
                    The fear of the Lord is the beginning of wisdom:
                    MODx Co-Founder - Create and do more with less.
                    • 28042 ☆ A M B ☆
                    • 24,524 Posts
                    Quote from: rthrash at Nov 15, 2005, 07:06 PM

                    How will this change with XML-RPC or AJAX calls, or will it at all for that matter!?!

                    Not a good idea to have basic site functionality depending on Javascript being enabled in the browser. Special webapps, sure, but not the general login forms or contact forms that would be affected by this.
                      Studying MODX in the desert - http://sottwell.com
                      Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
                      Join the Slack Community - http://modx.org
                      • 1764
                      • 680 Posts
                      IMHO I don’t think it’s possible for MODx in and of itself to be able to tell how to cache a snippet.

                      Checking the database is certianly not going to cover all of the bases. It won’t take care of external databases, date functions and other PHP only snippets, or snippets that get their output from text files, soap calls, etc.

                      The most "magical" way that I can think of doing this reliably is to have a cacheable flag associated with the snippet itself. I think this would clear up a lot of confusion with template designers. It still might be nice to have a way to override this in the tag, but I don’t think it’s necessary either.