Recursive parser

Hmm, tried to extract the logic from current parser, but I don’t get a clue on how the snippets are parsed currently, to be more precise, how the properties for a snippet are parsed and applied.

Due that, I’m not able to compare both, sorry.
(preg_match_all() seems to be a very fast function, so don’t expect the recursive parser to be faster though)

But don’t be affraid, the current parser allready _is_ fast!

I suppose it makes sense to have this in mind:

- What do we want the parser to be able to do.
- How is caching supposed to work.
- How can we make the caching smart and robust

In my profiles from the current parser, nothing showed up as bottlenecks in regard to the parsing logic (preg_match_all(), or strpos for example)

In my opinion, the truth behind a fast parser is not that it’s so fast that it can handle xxx requests per second, but that it only parses the parts that needs to be parsed.
So, in other words, use the cache if possible.

Right now, the [!Yada!] makes it possible to insert some dynamic piece of html code into a cached html page.

Some numbers from my poor server:

True cached page: 16 requests/sec
Cached with [!!] : 12 requests/sec
Non cached       : 10 requests/sec

As you seed, the evaluating of the snippets, and the corresponding executed php code,as well as the database calls degrade performance more then the parser does in respect to the little parsing time and replacement of [[ ]] tags.

However, the difference of a true cached non-cached an semi-cached page are not so huge!

So, what about adding "smart" logic to determine if a snippet has to be executed or not, and depending on the result, use true cached, semi cached or non cached template parsing?

For example, how many pages will actually change, even on a very dynamic site?
Once per minute?

Then the question is, how many page requests does this site get? Lets say, 60 per minute.
Then I wouldn’t be surprised to have the pages cached, and recreated after a page has been added/updated in the database, and still having a better speed increase, then using [!!] logic....
(I hope you still followed me )


(In other words, simply clean the cache after a page modification, and start caching the generated pages from that point again. But perhaps I’m overlooking something important now...)


Remon

Oops, double post!

I’m slowly starting to understand some of the logics of the parser, and some of those logics are ehm, well, not too smart ;-P

The recursive parser will have a great advantage over the current one, that it only has to parser the source ones, and not many times "just" to see if there are any snippets left.

I integrated the new recursive routine into the current parser, you might try it out

2 functions, of course the ’processTags()’ and another one to keep it readable (taken and modified from the parseSnippet function) ’getSnippetCode()’.


Look for processTags to see what changed!

I hope this will be a starting point, suggestions are of course welcome

Greetz,

Remon

P.S.
Fixed the php ’notice’ warnings while working on it too
Raymond: very nice routine you wrote!!!

Hi Remon,

Very nice work Did you get a chance to compare the new parser with the current one?

Many thanks

Will this solve the issue of using a snippet or TV as the argument value for a snippet? ( [[MySnippet?id=`[[AnotherSnippet]]`]] won’t work, you have to use the [!...!] form for the outer snippet)

Susan that’s exactly what it should do, wihtout using [! vs. [[ ...

Excellent! And thank you! I can pick around the edges and do little odds and ends, but that sort of thing is beyond me.

Quote from: xwisdom at Nov 14, 2005, 09:34 PM

Very nice work Did you get a chance to compare the new parser with the current one?

From a number of test runs, I’ve the feeling that the recursive parser is a bit less performing then the preg_match() function in the old parser.

However, in the old parser, there are 2 calls to parseDocumentSource(), which internally also can have multiple loops on the document to get some sort of recursive behaviour....

If coded correctly, the recursive parser really has to only parse the document source one time!

The [!!] are replaced by [[]] in the last stage of the parsing, so I run simply the new recursive snippet-parser after that, which results in only one call to parseDocumentSource().

And that of course results in better performance!! ;-)

I can’t go on now, since I don’t know how to proceed. parseDocumentSource() is also merging the chuncks, etc, and with a unified calling convention, this can be moved to processTags() too, but perhaps some more brainstorming has to be done on this issue?
(like partial caching, caching per object type, etc)

But for sure, this is gonna be much better then before

Remon

P.S.
Attached the modified parser to the first post of this thread!
Added a little modification, so the routine now continues after a snippet evaluation at it’s current parsing position, instead of starting at position 0 of the documentsource

Very nice

Do you have any figures (in milliseconds) to show the difference in time?

Quote from: xwisdom at Nov 15, 2005, 09:09 AM

Very nice

Do you have any figures (in milliseconds) to show the difference in time?

Thanks ;-)

Well, I couldn’t resist, and added the [[* *]], [[( )]], [[+ +]] and [[{{ }}]] too, though, I used the existing functions for them.
I suppose, thats fine, since chuncks and placeholders don’t have properties etc.

That makes one parsing of the source, which is very nice

You can find the modified parser in the first post!

I’ll now try to generate some profiling numbers ;-)

Remon

(P.S. Note that [!!] which means partial caching no longer works, only the above format is working, so if you like to test out, install the default modx installation, and change the template snippet etc calls to fit the above format, then you can test for yourself if it works (better))

A quick comparison between 2 default modx installation with example website installed, but one with the new parser routine and modified template of course.

Since [! doesn’t work with the modified parser, and somehow the caching doesn’t work correctly either (see comments in source code), I compared those 2 with

- cache disabled
- home document
- server (amd k6 400)
- no error callback function registered
- ab -n 100 (apache bench, concurency = 1, number of requests = 100)

Original parser:
~ 6.0 requests/second

Modified parser:
~ 7.0 requests/second

Looks good, no?

Greetz,

Remon