Charset latin1 Problems

Hey,
I’m encountering some strange issues with the Ajax Part of the Ajax Search and am hoping someone can point me in the right direction.

At http://www.trendlux.de/relaunch we have a search, but typing something like PR- into it, although it works, special characters such as Umlauts and the like are not being shown correctly.

The database is using the latin1 charset as is the the site_content table.

I would appreciate any pointers.

Cheers,
Taff

As far as I understand, you must use TinyMCE "Entity Encoding: Raw" (from plugin configuration) with AjaxSearch and content with special characters.

This is is not a problem with utf-8 encoding but with latin1 you may have a problem.

Hey,

thanks for the swift reply.

Could you walk me through where I need to insert that/update it please, it would be greatly appreciated?

I found it, and updated as you suggested, unfortunately to no avail.

The text it is displaying is in fact in the pagetitle and description fields, would that still be influenced by TinyMCE?

Thanks again for your help,
Taff

Quote from: Taff at Apr 22, 2009, 06:49 AM

The text it is displaying is in fact in the pagetitle and description fields, would that still be influenced by TinyMCE?

Then it’s not a problem with TinyMCE. This seems like one of those nasty database collation/connection problems. In your database - is for example letter "ü" displaying as "ü" or "ü"?

In the database itself it is being displayed as a ü. I used latin1, as opposed to my predecessor who had used a wild mixture of latin1_general, latin1_swedish, utf-8 and many others. I’ve spent quite some time hunting through the AjaXSearch JS and PHP includes trying to find out where I need a URIencode(), escape() or something else.

I have finally found the cause of all the problems:

ajaxSearchPopup.class.inc.php at ~ line 206 was calling to:



as it was presuming that I had the default UTF-8 charset/ collation. Beyonds as how I’m using latin1, this isn’t needed and simply commenting it out solved it.

I can now finish a happy (well as happy as you can be after spending a couple of hours on something like this).

On that note, thanks very much for your assistance, and hopefully this will save someone else time.

Taff

I’m sure this will be very useful information for others with similar problems.

Taff, the logic regarding the collation stuff is the following:

1/ first determine the charset of the database (ajaxSearchPopup.class.inc.php - line 218)

$this->dbCharset = $database_connection_charset; // database charset (search.class.inc.php - line 77)

2/ I make the supposition that your page charset is coherent with your database charset

In your case, as i understand, your database charset is latin1, $this->pageCharset[’latin1’]) give a pageCharset = ’ISO-8859-1’

3/ If the page charset is different of UTF-8, first I convert the searchstring to be coherent with the database charset (ajaxSearchPopup.class.inc.php - lines 300 to 303):

4/ As the result from the database is encoded in "latin1" a conversion to "UTF-8" is required before to output in the ajax window (UTF-8 required). So as $this->needsConvert = true finally you need to convert your result before output to the window (ajaxSearchPopup.class.inc.php - line 206)

I have a French site, with a latin1 encoded database and ’ISO-8859-1’ encoded html pages. And I have no issues. I could search terms with accented letters.

Hey,

yeah I’m still a little lost as to why it works as intended when commenting out that line.

I am indeed using a pageset of iso-8859-1 and a database collation of latin1.

I’m just happy that I can move on the the next on the list and maybe someone else can benefit from the information I provided

Taff