SimpleSearch Chinese encoding issue

45 Posts

Beniamin Reply #1, 11 years, 6 months ago

Hello all,

I'm doing a site in Chinese (Revo 2.2.2), and have had no problems until I got to implementing SimpleSearch (1.6.0). On the search results page, each of the previews just shows a series of question mark diamonds (�) instead of the proper Chinese characters.

When I click on one of the results, the page displays correctly, so this must be a problem with SimpleSearch. I've looked high and low for clues, and I'm still clueless.

Any ideas? Need any more information from me?

Thanks!

2 Posts

Yossi Reply #2, 11 years, 5 months ago

Hi,

Did you fixed your encoding issue?
I have the same problem with Hebrew site (Revo 2.2.5-pl)

Thanks!

45 Posts

Beniamin Reply #3, 11 years, 5 months ago

Not to my satisfaction, although I was able to find a workaround. It has something to do with the encoding within the addHighlighting() function, but I wasn't successful in changing the encoding there. Instead I just bypassed the function and got my results with no highlighting, but at least intact. So if you aren't using highlighting, this will work fine for you; otherwise, you'll have to keep digging. And if you do figure something further out, please let me know!

OK, here's what I did:

I edited the SimpleSearch snippet (i.e. the snippet with that exact name), and replaced this line:

$resourceArray['extract'] = !empty($highlightResults) ? $search->addHighlighting($extract,$highlightClass,$highlightTag) : $extract;

...with this:

$resourceArray['extract'] = $extract;

Basically giving the original search result without the condition, and without calling the addHighlighting() function.

Hope this helps!

2 Posts

Yossi Reply #4, 11 years, 5 months ago

Thanks! I took your lead and found a solution.

In simplesearch.class.php (core\components\simplesearch\model\simplesearch) change addHighlighting function as follow:

 $string = preg_replace('/' . $quoteValue . '/i', '<'.$tag.' class="'.$cls.'">$0</'.$tag.'>', $string);

with this one:

$string = mb_ereg_replace('/' . $quoteValue . '/i', '<'.$tag.' class="'.$cls.'">$0</'.$tag.'>', $string);

This works fine for me and you don't need to bypass the addHighlighting function

☆ A M B ☆
24,524 Posts

sottwell Reply #5, 11 years, 5 months ago

Unfortunately this can't be a global core change for advsearch since many hosts don't provide the mbstring library for their PHP. http://php.net/manual/en/mbstring.installation.php

Even if the hosting does provide this, you can very easily implement it for snippets that don't use it with function overloading http://php.net/manual/en/mbstring.overload.php

Studying MODX in the desert - http://sottwell.com
Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
Join the Slack Community - http://modx.org

6 Posts

borys Reply #6, 10 years, 8 months ago

Hi there,

I came across the same issue and found hack-fix solution which works well for me (some of Czech national characters were wrong within search results extracts from SimpleSearch snippet)

In '/core/components/simplesearch/model/simplesearch/simplesearch.class.php' (about line 300) I changed:

public function createExtract($text, $length = 200,$search = '',$ellipsis = '...') {
    $text = trim(preg_replace('/\s+/', ' ', $this->sanitize($text)));
    if (empty($text)) return '';
    ...

into this:

public function createExtract($text, $length = 200,$search = '',$ellipsis = '...') {
    if ($this->modx->getOption('use_multibyte',null,false) && function_exists('mb_ereg_replace')) {
        $text = trim(mb_ereg_replace('/\s+/', ' ', $this->sanitize($text)));
    } else {
        $text = trim(preg_replace('/\s+/', ' ', $this->sanitize($text)));
    }
    if (empty($text)) return '';
    ...

Hope this helps...

24,544 Posts

BobRay Reply #7, 10 years, 8 months ago

This may not be the issue, but it's worth checking. Look at the character set of all tables *and* text fields in the MODX DB with PhpMyAdmin to make sure it matches the character set in the MODX core/config/config.inc.php file.

If not, see this: http://bobsguides.com/convert-db-utf8.html

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting