Readme.txt display chokes on French accented character

A thank you to René Tschannen in my readme.txt causes nothing to be displayed after the n in René during the install. The rest of the file doesn’t display.

The missing text is all there in the DB and the character looks as it should there.

What’s throwing me is that everything I can think of is set to UTF-8.

The DB charset
The DB collation charset
The charset of the table
The charset of the field in the table
The MODx charset
The editor I use to edit the readme.txt file

When I view the source of the Manager page, the charset is set to UTF-8.

BobRay,

I put René Tschannen into a readme.txt for a test package, and can’t seem to get it to throw any errors. It works fine on my install.

No clue.

Quote from: splittingred at Nov 14, 2008, 10:12 AM

BobRay,

I put René Tschannen into a readme.txt for a test package, and can’t seem to get it to throw any errors. It works fine on my install.

[
Attached (The name is near the end). This is the file I used in my build. I had to rename it to get it to upload here.

This is making me nuts.

I looked at the file in a hex editor and the character is hex E9 (decimal 233) which is the correct UTF-8 character code. When I do an SQL-dump of the transport_packages table (dumping as UTF-8), the character appears as the two-byte sequence C3 A9. It still looks correct when that file is loaded in my editor or Wordpad but gets munged in notepad.

The original file, BTW, looks fine in notepad, wordpad, and when opened directly with FF, IE, or Chrome.

It appears that the character is being converted either by MODx (which seems unlikely) or by MySQL when it gets put into the DB.

Any clues appreciated.

I’m getting nowhere with this. I’ve confirmed that the character is an E9 in the manifest and in the manifest in the transport.zip file. I would assume that during the install, MODx gets the readme content directly from the manifest rather than the DB so the DB settings shouldn’t really matter.

On the theory that using utf8_unicode_ci instead of utf8_general_ci might be causing a problem, I created a new install with the everything set to utf8_general_ci and did a build and install in that seting but it made no difference.

I can paste the name René into a document’s content and it shows fine in the Manager.

I’ve checked and re-checked everything and ALL my MySQL settings are UTF-8 as are all my MODx settings. I tried both utf8_unicode_ci and utf8_general_ci.

The first-choice font for the panel is Courier New, which definitly supports the E9 character.

Looking at the MODx code hasn’t gotten me anywhere (a search for "encode" turns up over 500 entries, for example). I did find code in json.php that has functions for converting back and forth between utf-8 and utf-16, but neither function seems to execute during a build or an install. Looking at the page source is no good since it’s all done in js. I’m afraid that René is just going to have to be referred to as Rene unless I can figure this out.

BobRay - I know that JSON (and JavaScript) in general has problems with non-UTF-8 characters, but I’m not sure why it trips on that one. I’ll look into it this week and see what I can find.

Quote from: splittingred at Nov 17, 2008, 10:42 PM

BobRay - I know that JSON (and JavaScript) in general has problems with non-UTF-8 characters, but I’m not sure why it trips on that one. I’ll look into it this week and see what I can find.

Thanks! I suspected that at first, but E9 is the correct code for the é character in both Latin1 and UTF-8. The question is, how and where is that code being converted to C3 A9 and why would MODx refuse to show the rest of the file after that sequence rather than just showing those characters (Ã and © -- both legit and equal in both Latin1 and UTF-8).