We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 27622
    • 17 Posts
    i think i found the solution on my own, switching the following lines:
          // clean the text if needed
          $text = $this->cleanText($text,$this->cfg['stripOutput']);
          // get the extract
          $text = $this->getExtract($text,$this->searchString,$this->advSearch,HIGHLIGHT_CLASS,$nbExtr);
    

    in
      function addExtractToRow($row)
    to
          // get the extract
          $text = $this->getExtract($text,$this->searchString,$this->advSearch,HIGHLIGHT_CLASS,$nbExtr);
          // clean the text if needed
          $text = $this->cleanText($text,$this->cfg['stripOutput']);
    

    this way the special characters get encoded AFTER the extract is taken... so you can’t cut off halfway encodings
    i hope this has no negative effects... maybe someone else can verify this?
      • 27622
      • 17 Posts
      i was to early with my conclusion... this encodes chars correctly but strips html after taking an extract... so when extracting halfway a html tag the html tag doesn’t get extracted... i think i can’t avoid to do the stripping before taking extract end using a new function for encoding special chars after that...

      ok i don’t need a function to do this but i should do it after taking extract...
      so i left stripoutput as it were but changed addExtractToRow like this:
        function addExtractToRow($row){
      
          $text = '';
          $nbExtr = 0;
      
          if ($this->extractNb) {
            // From row get all the fields allowed to use for extract
            foreach($this->extractFields as $f) if($row[$f]) $text .= $row[$f] . ' ';
      
            // clean the text if needed
            $text = $this->cleanText($text,$this->cfg['stripOutput']);
      	  
            // get the extract
            $text = $this->getExtract($text,$this->searchString,$this->advSearch,HIGHLIGHT_CLASS,$nbExtr);
      
            // html_entity encoding
            $text = htmlentities($text,ENT_QUOTES,'UTF-8');
      
          }
          $row['extract'] = $text; // set the concatened extracts
          return $row;
        }
        • 27622
        • 17 Posts
        and i post before testing and that’s a mistake because it doesn’t work...
        someone know’s what works?
          • 27622
          • 17 Posts
          after some more little changes I can’t remember exact... I cant reproduce this error so I’m hoping it’s gone...
          and if it is gone I thank you for your help and I hope it improves the performance of new releases of ajaxsearch...
            • 27622
            • 17 Posts
            there is one (i hope) last problem... when searching for "op" (above in dutch) also © is in the results and gets highlighted... so the code becomes..
            &c<span class="ajaxSearch_highlight ajaxSearch_highlight4">op</span>y;
            so it becomes invalid...
            i hope someone finds some solution for this... (i’m going to sleep now wink )
            it can wait until tomorrow
              • 5811
              • 1,717 Posts
              I tried a little bit and found that storing raw chars for é, á, ç and so on is not a big problem, even € and © work fine but the often used & and < give problems... and there are maybe more...
              I’m not sure but probably all five predefined xml character entity’s ( &, <, >, " and ’ ) give problems..

              From http://wiki.modxcms.com/index.php/TinyMCE for tinyMCE configuration:

              Entity Encoding
              This option controls how entities/characters gets processed by TinyMCE. The value can be set to numeric, named or raw. Where numeric is numeric representation such as "&#160;" named is entity names such as "& " and raw is " ". The default value of this option is named, if named is used the entities option will be used to convert the codes into names.

              Encoding types:

              named

              Characters will be converted into named entities based on the entities option. For example, a non-breaking space could be encoded as &nbsp;.

              numeric

              Characters will be converted into numeric entities. For example, a non-breaking space would be encoded as &160;.

              raw

              All characters will be stored in non entity form except these XML root entities: & = < > " ’

                • 27622
                • 17 Posts
                Quote from: coroico at Nov 18, 2009, 12:04 PM

                Entity Encoding
                This option controls how entities/characters gets processed by TinyMCE. <cut>
                Thank you for reminding me of this, but I don’t see how this could help me...
                To summarise: my stripOutput function looks like this at the moment:
                  function StripOutput($text){
                    if ($text !== ''){
                      // replace line breaking tags with whitespace
                      $text = stripLineBreaking($text);
                      // strip modx sensitive tags
                      $text = stripTags($text);
                      // strip Jscripts
                      $text = stripJscripts($text);
                      // strip html tags. Tags should be correctly ended
                      $text = stripHTML($text);
                      // html_entity encoding
                      $text = htmlentities($text,ENT_QUOTES,'UTF-8');
                    }
                    return $text;
                  }

                This solves the problem of the output of unencoded special characters (like &-sign).

                But gives two new problems

                The first is when searching a document containing the following string of testing code:
                <p>Zoek & &apos; ' © één één Curaçao kun je  zoekéén ></p>

                when searching for the word "Zoek" (search in Dutch) the result looks like this:
                  <div class="ajaxSearch_resultExtract"><p>...<span class="ajaxSearch_highlight ajaxSearch_highlight1">Zoek</span> & &apos; ' © één één Curaçao kun je  <span class="ajaxSearch_highlight ajaxSearch_highlight1">zoek</span>é&eacute...</p></div>

                The problem is in the end where the line gets cut of halfway the encoding of a character and making the code invalid because the ";" is missing.

                The second problem is this: when searching for "amp" (without the quotes) the results contain pages with "&amp;" in it which represents the &-sign... but highlights it like this:
                &<span class="ajaxSearch_highlight ajaxSearch_highlight1">amp</span>;
                so the code gets invalid again...
                  • 27622
                  • 17 Posts
                  OK (I think) I finally solved both problems... not by editing AjaxSearch (I also restored the defaultStripOutput function) but by using The XHTML+XML+Unicode 1.0 plug-in by Xsss4hell.
                  So AjaxSearch outputs raw characters (solving the breaking problem halfway chars) and the XHTML+XML+Unicode Plug-in encodes special characters as an "OnWebPagePrerender" system event.

                  I’m going to stress-test this combination a little bit, when I can’t find any more related problems I will mark this thread as solved...

                  But I think the author(s) of AjaxSearch should keep an eye on this issue and maybe try to solve the incomparability with special character encoding (named entity’s) maybe the approach of the above named plug-in could help...
                    • 27622
                    • 17 Posts
                    Quote from: SlechtValk

                    OK (I think) I finally solved both problems...
                    Again I was to quick with my conclusions, the problem is solved in the search results, but not in the linked pages.
                    &<span class="ajaxSearch_highlight ajaxSearch_highlight1">amp</span>;

                    Although this is probably not a problem of AjaxSearch but of the used highlighting plug-in (Search_Highlight v1.3).

                    Should I start a new topic (outside of the ajaxsearch subforum) or is search highlighting so much entangled in ajaxsearch it should remain here?