We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 27622
    • 17 Posts
    I want to use AjaxSearch in my website, but I have some pages with & and € in it, but AjaxSearch doesn’t encode special characters in the results.
    While I’m serving as "application/xhtml+xml", browsers crash on every character that is not encoded with an XML-parse-error.

    Is there a way to make AjaxSearch encode its results well?
      • 5811
      • 1,717 Posts
      I want to use AjaxSearch in my website, but I have some pages with & and € in it,
      Do you use TinyMCE with the entity_encoding parameter set to "named" rather than "raw" (see the configuration tab of TinyMCE). Or these characters comes from an other source ?

      Is there a way to make AjaxSearch encode its results well?
      Look at the &stripOutput parameter, to add a specific encoding before output the results:

      Extract of the ajaxSearch_readme.txt:
      ---- &stripOutput user function    (optional)
              to transform on fly the result output
              by default: defaultStripOutut
      
              StripOutput user function should be define in the config file as follow :
      
              // string functionName(string $text)
              // functionName : name of stripOutput function passed as &stripOutput parameter
              // $text : string php variable name as results
              // return the filtered results
              /*
              function myStripOutput($text){
      
                  Any Php code which filter the results
                  The following internal functions could be called:
                    $text = stripTags($text); // strip all the MODx tags
                    $text = stripJscript($text); // strip jscript
                    $text = stripLineBreaking($text); // replace line breaking tags with whitespace
                    $text = stripHtml($text); // strip all the html tags
      
                  You could also developp you own filter based on regular expressions.
                  See http://fr.php.net/manual/en/intro.pcre.php
      
                return $text;
              }
              */
      
              By default : defaultStripOutput function will be used if &stripOutput parameter
              is not set or if the function is not defined :
      
              function defaultStripOutput($text){
      
                // replace line breaking tags with whitespace
                $text = stripLineBreaking($text);
                // strip modx sensitive tags
                $text = stripTags($text);
                // strip Jscript
                $text = stripJscripts($text);
                // strip html tags. Tags should be correctly ended
                $text = stripHTML($text);
      
                return $text;
              }
      
        • 27622
        • 17 Posts
        Coroico thank you for your reply, bud I still have some questions.

        Quote from: coroico at Nov 17, 2009, 03:45 AM
        Do you use TinyMCE with the entity_encoding parameter set to "named" rather than "raw" (see the configuration tab of TinyMCE). Or these characters comes from an other source ?
        No I don’t use TinyMCE at all (I have it installed but don’t use it), I mostly write my code by hand... but this is not the problem because I want to keep using named entity character encoding. This encoding works well for me in the rest of the site, and I know what these codes stand for... (I don’t know the raw or numerical encoding by heart...).

        Look at the &stripOutput parameter, to add a specific encoding before output the results:
        How would this help me? (my knowledge of PHP is not that good that I could rewrite this function) is there a special function for this or should I just str_replace all special characters to there encoded form?

        Edit: OK I found a way i added the following to my function StripOutput:
        	  // encode special characters
        	  $text = htmlspecialchars($text);

        But now a new question rises, because I can’t oversee what the effect could be...
        Could this have any negative effect? Characters that shouldn’t be encoded that get encoded or something like that?
          • 5811
          • 1,717 Posts
          I don’t know the raw or numerical encoding by heart...
          The raw encoding leave the characters as typed. e.g: & is stored as & in the database and € stored as €
          With the named entity configuration, when you type &, the character is stored as & and € is stored as €

          This is a key point for the search, for accented languages. When you search "été" (summer in French) you search "été" not "ét&eacute"
          is there a special function for this or should I just str_replace all special characters to there encoded form?
          As the characters coming from the database are not transformed, you got & € and possibly all the named entity characters. So you need to encode the output. For this use the html_entity decode function.
          http://www.php.net/manual/en/function.html-entity-decode.php. But as I haven’t tested, I am not sure of the result ...
            • 27622
            • 17 Posts
            Thanks for your explanation about raw storage of characters (I didn’t know that, thought it stored it like hex codes).

            Quote from: coroico at Nov 17, 2009, 11:32 AM
            When you search "été" (summer in French) you search "été" not "ét&eacute"
            Ok That raises a fair point... The changes I made (and mentioned in my edit) did the trick for well formed xml but for searching it didn’t work well...
            but when I use an untouched version of the functions and I search for "één" (one in Dutch) I don’t get any results at al... (while I’m sure that’s used at least once).

            So do you think I should store special characters unencoded/raw (for searching purposes)? (that feels really strange... writing (x)html without encoding special chars)
            But how do I tell modx it should encode these chars at the last moment before showing it?
              • 5811
              • 1,717 Posts
              Go on this demo page, choose French documents in the left menu, and type "été" as search term. You got a page with several accented characters. They are encoded (see the code of the page) as they are stored in the utf8 database.
                • 27622
                • 17 Posts
                Quote from: coroico at Nov 17, 2009, 01:17 PM

                Go on this demo page, choose French documents in the left menu, and type "été" as search term. You got a page with several accented characters. They are encoded (see the code of the page) as they are stored in the utf8 database.
                yeah but when I do this and validate the code it gives more than 50 errors and about the same number of warnings (22 Errors, 21 warning(s) when no search is performed yet)... mostly &-signs give trouble...
                you are using "XHTML 1.0 Transitional" I’m trying to use Strict and until I tried to implement a search-function (which on itself is doing it’s job ok except for the output) Strict works fine for me.

                Take a look at the W3C validator and you’ll see the first error given (at the moment of posting) is related to an unencoded &-sign. (character "&" is the first character of a delimiter but occurred as data) and there are lots more almost all related to misuse of the &-sign.
                  • 5811
                  • 1,717 Posts
                  You are right. Check this new demo, and let me know if you prefer this one ...

                  I have applied a xhtmlStripOutput function on the output. Here is the content of this function:
                  <?php
                  
                  /**
                   * xhtmlStripOutput : xhtml ouput strip function
                   */
                    function xhtmlStripOutput($text){
                  
                      if ($text !== ''){
                        // replace line breaking tags with whitespace
                        $text = stripLineBreaking($text);
                        // strip modx sensitive tags
                        $text = stripTags($text);
                        // strip Jscripts
                        $text = stripJscripts($text);
                        // strip html tags. Tags should be correctly ended
                        $text = stripHTML($text);
                        // html_entity encoding
                        $text = htmlentities($text,ENT_QUOTES,'UTF-8');
                      }
                      return $text;
                    }
                  
                  ?>
                  The last function used is htmlentities (see: http://www.php.net/manual/en/function.htmlentities.php Which is curious is that some accented characters remain inchanged.

                  For the demos, if you check with the W3C validator, it remains lot of errors ( tongue), but I haven’t still found the time to solve all the errors listed. May be one day ...
                  As it is a demo site, I don’t spend too much time on this. May be with some help ...
                    • 27622
                    • 17 Posts
                    I tried a little bit and found that storing raw chars for é, á, ç and so on is not a big problem, even € and © work fine but the often used & and < give problems... and there are maybe more...
                    I’m not sure but probably all five predefined xml character entity’s ( &, <, >, " and ’ ) give problems...
                    tested some more and strangely enough from the above five only & and < seem to be giving problems... (probably because they are opening signs for special tags) but i am not sure if these are the only ones...
                    still it doesn’t feel right to use unencoded ©, €, é and so on in xhtml...

                    your demo (id=375) still gives errors but after a closer look most of them are in the menu ( [title="Debug with FirePhp & FireBug"] is one of them) and the URL’s... so the searchresults look fine... (except for &’s in URL’s but i’m not using them at the moment)

                    And now the question is obviously, how do I accomplish this?


                    (i’m guessing saving chars raw... but i think i have to change some outputting code am i right?)

                    edit: i didn’t read your full post and didn’t see the code i’ll look into that soon
                      • 27622
                      • 17 Posts
                      OK, I’ve looked into it a little bit and this change seems to do the trick... with one exception (found by accident)...

                      i’m using the following test-string: "Zoek & ’ ’ © Hier één Curaçao kun je € zoekéén >" encoded like this:
                      Zoek & &apos; ' © Hier één Curaçao kun je € zoekéén >

                      (some Dutch text with some testing characters) it encodes it well but....

                      the search result is this:
                        <blockquote class="ajaxSearch_resultExtract"><p>...<span class="ajaxSearch_highlight ajaxSearch_highlight1">Zoek</span>en 
                      <span class="ajaxSearch_highlight ajaxSearch_highlight1">Zoek</span> & &apos; ' © Hier één Curaçao kun je € <span class="ajaxSearch_highlight ajaxSearch_highlight1">zoek</span>é&eac...</p></blockquote>
                      

                      It gets cut off halfway the character encoding so in the end I still get an error... (I agree this is a rare case but it just happened to me...)

                      maybe this re-encoding should happen after the string gets cut off... (but i’m not sure where i should do this)