illegal character encoding - mb_convert_string issues

228 Posts

christianhanvey Reply #1, 16 years, 2 months ago

I can’t get any results to show up since upgrading from 1.8.4 to 1.9.0 packaged with my MODx 1.0.4 upgrade.
When I type in a search term, I’m getting a js error in my firebug panel:

invalid regular expression flag h
line 25: /assets/snippets/ajaxSearch/js/ajaxSearch1/ajaxSearch-jquery.js

If I inspect the post response from the search it shows a php error in the results

<br />
<b>Warning</b>:  mb_convert_encoding() [<a href='function.mb-convert-encoding'>function.mb-convert-encoding</a>]: Illegal character encoding specified in <b>/home/tresilli/public_html/assets/snippets/ajaxSearch/classes/ajaxSearchOutput.class.inc.php</b> on line <b>238</b><br />
{"res":" <p class=\"AS_ajax_resultsInfos\">1 result found for \"<span class=\"ajaxSearch_highlight ajaxSearch_highlight1\">career</span>\".<span class=\"AS_ajax_resultsDisplayed\">Displayed: 1</span></p> <div id=\"grpResult_site_wide\" class=\"AS_ajax_grpResult\">  <div class=\"AS_ajax_grpResultName\"> <span class=\"ajaxSearch_grpResultsDisplayed\">(Displaying 1 - 1 of 1)</span></div>  <div class=\"AS_ajax_result\">   <a class=\"AS_ajax_resultLink\" href=\"/recruitment/why-a-career.html?searched=career&advsearch=oneword&highlight=ajaxSearch_highlight+ajaxSearch_highlight1\" title=\"Why work  \">Why work with Us</a>     <div class=\"AS_ajax_resultExtract\"><p>... We offer a unique <span class=\"ajaxSearch_highlight ajaxSearch_highlight1\">career</span> path in child and family health. Our Centres are located in excellent locations around ...</p></div>   </div> <div class=\"paging1\">  <span class=\"paging1Infos\"> 1 - 1 / 1 </span>  </div> </div>   ","resnb":"1,1","ctgnm":"[\'site_wide\']","ctgnb":"[1]"}

So it looks like some results are coming through, but they are not being displayed because theres some characters in there that the javascript parsing routine cant handle, most likely from the php generated warning. I’m not sure where to go from here. The line the php error refers to in ajaxSearchOutput.class.inc.php is:

if ($this->_needsConvert) $outputResults = mb_convert_encoding($outputResults,"UTF-8",$this->pgCharset);

Does this mean that the php function with that encoding is not available on my server?

Here is my current ajaxsearch call - i already load the jquery library 1.4.2 earlier in the page:

[!AjaxSearch? &showIntro=`0` &jscript=`jquery` &addJscript=`0` !]

AJAXSEARCH 1.9.0
MODX 1.0.4
PHP 5.2.13
LINUX/APACHE

☆ A M B ☆
24,524 Posts

sottwell Reply #2, 16 years, 2 months ago

No, you have it, it’s just throwing a warning because it doesn’t like your pgCharset.

It appears that when dealing with an unknown "from encoding" the function will both throw an E_WARNING and proceed to convert the string from ISO-8859-1 to the "to encoding".

Studying MODX in the desert - http://sottwell.com
Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
Join the Slack Community - http://modx.org

228 Posts

christianhanvey Reply #3, 16 years, 2 months ago

When i chuck a die statement in to check the value of [tt]$this->pgCharset[/tt], its value is empty and its type is NULL

Looking at the class itself, the property $pgCharset is not defined, and the class does not inherit from another. However, it is a property of the AjaxSearchConfig class. I also notice pgCharset is referenced in the form $this->asCfg->pgCharset, so I tried that in the ajaxSearchOutput class, and lo-and-behold it works now.

Hey, did I just find a bug?

1,717 Posts

coroico Reply #4, 16 years, 2 months ago

$pgCharset is defined as $this->asCfg->pgCharset. And $pgCharset is defined in ajaxSearchConfig class (function _setCharset, line 99):

//! initialize the page charset
$this->pgCharset = array_key_exists($this->dbCharset, $this->_pageCharset) ?         $this->_pageCharset[$this->dbCharset] : $this->dbCharset;

I think an error has been returned but has not been catched when the pgCharset has been defined.

Could you provide which page charset you use (UTF-8 or iso-8859-1 or ...) and check that the database charset is defined in your MODx config file. In the above code lines, $this->dbCharset comes from the MODx config file (see line 36):

        $this->dbCharset = $modx->db->config['charset']; //! database charset

228 Posts

christianhanvey Reply #5, 16 years, 2 months ago

the issue is with line 238 in the ajaxSearchOutput class:

if ($this->_needsConvert) $outputResults = mb_convert_encoding($outputResults,"UTF-8",$this->pgCharset)

The ’from encoding’ argument, i think, is wrong.
It should be called as [tt]$this->asCfg->pgCharset[/tt] rather than [tt]$this->pgCharset[/tt].
Changing it resolved the issue for me.

For the record, my manager/includes/config.inc.php file says:

$database_connection_charset = 'latin1';

and in my html templates I have:

<meta http-equiv="Content-type" content="text/html;charset=UTF-8">

1,717 Posts

coroico Reply #6, 16 years, 2 months ago

The issue comes from that you have a database connection charset = latin1 with a page charset with UTF-8.

This case is not managed by the AjaxSearch code. See ajaxSearchConfig class line 30:

    // Conversion code name between html page character encoding and Mysql character encoding
    // Some others conversions should be added if needed. Otherwise Page charset = Database charset
    var $_pageCharset = array('utf8' => 'UTF-8', 'latin1' => 'ISO-8859-1', 'latin2' => 'ISO-8859-2');

228 Posts

christianhanvey Reply #7, 16 years, 2 months ago

uh - ok. Understanding charsets is definitely not something i am totally clear about.
And I’m not sure what other way I should be going about resolving this.

So you are saying there isn’t a problem with the code on the line I have mentioned?
If not, Im confused as to why when I change it, that my issues go away...
And also why the output class property ’pgCharset’ is not defined, and hence has a value of NULL?
I see that pgCharset is a property of the config class, but the output class is not an extension of the config class so it is not inherited. Sorry if I am misunderstanding something!

If I change my html page charset to ISO-8859-1, matching the database encoding, are you saying that this would get round the problem??
EDIT: I tried that - it still gave me the Illegal character encoding error as mentioned in my first post.

1,717 Posts

coroico Reply #8, 16 years, 2 months ago

Code:
if ($this->_needsConvert) $outputResults = mb_convert_encoding($outputResults,"UTF-8",$this->pgCharset)

The ’from encoding’ argument, i think, is wrong.
It should be called as $this->asCfg->pgCharset rather than $this->pgCharset.
Changing it resolved the issue for me.

Apologize Crunch, you are totally right, this is a bug. $pgCharset is not defined for the class AjaxSearchOutput so $this->asCfg->pgCharset should be used instead of $this->pgCharset.

This issue is now registered as AJAXSEARCH-73
Thanks for this feedback.

228 Posts

christianhanvey Reply #9, 16 years, 2 months ago

no problems! now we both have something resolved!