0.9.6.2-rc2. problem with russian encoding after install

No checking out the latest trunk version in the 096 branch. That’s where development occurs. Last commit was 4500 I believe dealing with Resource Browser URL issues. Thanks!

I have tested and retested the encoding issues with the latest from SVN and it is working flawlessly as far as I can tell. If you were using MODx previously in an environment where your MySQL client connection was latin1, but you were using UTF-8 on the site and utf8 collations on your database, then when MODx was fixed to properly use SET NAMES or SET CHARACTER SET, all the data you previously saved that was being translated back and forth between latin1 and utf8 is now no longer being converted. Now you have a problem. So you will need to fix your content in the database, figure out what your actual content is being stored as in the database, or stick with the older versions.

@Fuzzy: None of those other statements are necessary; you just need to figure out what your configuration was previously and find a way to make it work now that things are set properly. Please read http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html for more information how the SET NAMES and or SET CHARACTER SET usage works.

Ryan, I have just tested the commit 4500 and I’m very glad to say: it works very well!

It’s really good to know that we will not have these problems in the future because there are many people (not only Russians, I know, for example, one Swedish man too) who has always the same issues with encoding.

I have attached two screenshots.

First screenshot shows the real problems with encodings if we install MODx 0.9.6.2 that was downloaded from official repository.
Second screenshot shows that all is working very good on the latest SVN version (4500 commit).

These tests I have done on my PC with Apache 2.0 and MySQL 5, default database encoding is UTF-8, OS Windows XP.





P.S.: I can’t upload my screenshots (they are about 300 kbytes in total).  I have attached them as external images.
The system said "The upload folder is full. Please try a smaller file and/or contact an administrator." Can it be fixed?

By the way, both of these versions (current 0.9.6.2 and 4500 commit) have a small issue with installations. If you try to change the language when you install MODx, for example, to Russian, German or any other language that has own alphabet symbols, you will see smth like that:

Добро пожалова - Russian
Sie müssen Ihren vollständigen - German
MODx�インストールを開始���〠- Japanese
and so on...

Of course this is not right language symbols.

It is happening because the webserver does not return right headers.
It should be "Content-Type: text/html; charset=utf-8" but really it returns "Content-Type: text/html; charset=ISO-8859-1".
So the browser ignores the META tag <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> and use the headers that it has received from the webserver.

Thanks for you Feedback Fuzzy!

I did not use a fresh install but a update of an existing site, maybe that is the cause. Also I am still wondering why an update wouldn’t work as I did not try to convert old but to insert new russian content.

Yesterday I tried upgrading a 0.9.6 utf8 utf8_unicode_ci install to 0.9.6.2-rc2. (The site happens to be written in Japanese.) The first time I did this, I used the standard upgrade path, and I experienced the same garbled text encoding problem in the manager as described here. I checked the config.inc.php file and the character set was correctly specified as ’utf8’ and the "character set" method was also set. I couldn’t see any info about collation.

So, I tried a second time, but this time I chose the advanced upgrade option, which let me specify the character set and collation myself. MODx had pre-filled in all the fields with the correct information, apart from the collation, which was set to utf8_general_ci. I changed it to utf8_unicode_ci, proceeded with the installation and it all worked fine.

My conclusion is that MODx was not determining the database collation correctly. Perhaps it didn’t have the information and just guessed utf8_general_ci? If so, it would be good if a warning and the option to change the collation from utf8_general_ci on the standard upgrade was provided.

Thanks.

Hi everyone!

After trying different things to enable Russian characters with my 0.9.6.2 MODx installation I finally found a solution. I did a backup of the database with the MODx backup tool and hat a look at it with a text-editor. I discovered that although the database was set to UTF-8 and the content was also UTF-8 encoded the tables still where to be created with Latin1:



All I did was to replace all ’CHARSET=latin1’ to ’CHARSET=utf8’ and reimport the database dump. After that little tweak Russian characters worked all of a sudden even with 0.9.6.2.

Maybe this info is helpful for those wo have 0.9.6.2 and don’t want to make a fresh install and take over all the content manually because of encoding problems.