Encoding problem upgrading from modx-0.9.6 to modx-0.9.6.1p2

You need to find out what character set/encoding the database you have MODx installed in is using, and make sure the tables inside it have a collation that matches. If these are not utf8, then you cannot use utf8 or it will try converting everything from whatever encoding you have to UTF-8 automatically as part of the mysql client connection.

That’s the thing. They are set correctly. For the database I get...

show variables like "character_set_database";
character_set_database utf8

show variables like "collation_database";
collation_database utf8_unicode_ci

... and for the tables I get, for example (showing only encoding related info for brevity)

SHOW CREATE TABLE `so_site_content`;

CREATE TABLE `so_site_content` ... `type` varchar(20) collate utf8_unicode_ci ... `contentType` varchar(50) collate utf8_unicode_ci ... `pagetitle` varchar(255) collate utf8_unicode_ci ... `longtitle` varchar(255) collate utf8_unicode_ci ... `description` varchar(255) collate utf8_unicode_ci ... `alias` varchar(255) collate utf8_unicode_ci ... `link_attributes` varchar(255) collate utf8_unicode_ci ... `published` ... `introtext` text collate utf8_unicode_ci ... `content` mediumtext collate utf8_unicode_ci ... `menutitle` varchar(255) collate utf8_unicode_ci ... DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ...


Paul

I recently moved all my sites when I upgraded my server. In the process of doing this I had to export and re-import my modx databases. I encountered the same encoding problem, so I realised that this was not an upgrade issue. In fact, the problem was that I had been using phpmyadmin to do the export. I don’t know if I just didn’t choose the right options when doing the export, or if the version of phpmyadmin that I was using just has a buggy export facility, but it was breaking the encodings somehow.

I just used the standard mysqldump command and that fixed all problems. Conclusion: Don’t rely on phpmyadmin to correctly export utf-8 encoded websites.

Cheers,

Paul

Without exporting/importing the database I experienced that $database_connection_charset = ’utf8’; tries to read 8 bit characters. I suppose it also writes 8 bits.
For me it’s only working with $database_connection_charset = ’’; -- so if it does, I think it’s a bad habit to change this setting during the upgrade process.
(My database collations in different installs are either utf8_general_ci and utf8_unicode_ci.)

I’ve read this before about phpMyAdmin’s export function.

For me it’s only working with $database_connection_charset = ’’; -- so if it does, I think it’s a bad habit to change this setting during the upgrade process.

Some snippets require the charset to be set, like AjaxSearch (1.8). I see your point about setting this during update though.

Interesting about the phpmyadmin issue, I wonder if that is what gave me all sorts of issues on a previous project where I did export/import via phpmyadmin. Good to know.

I have had the same encoding problem.

Testing a website locally, with UTF-8 set. Then moved the website to 0.9.6.2 and it all went astray.

I had two backups : one made with PHPMyAdmin, and the other by MODx. None of them two worked.

I have removed UTF-8 from $database_connection_charset and it now works.

What I did was a fresh install of MODx instead of an upgrade and the $database_connection_charset parameter was set to UTF-8. In the original config.inc.php, it was set to ’’.
My two backup files were made with settings : $database_connection_charset=’’

It is thus important to pay attention to this parameter when upgrading or moving from a server to another.

Could you repeat your tests with the recently-posted 0963 RC1?

http://modxcms.com/forums/index.php/topic,29689.0.html

Quote from: kilroy at Oct 16, 2008, 12:15 PM

I have had the same encoding problem.

Testing a website locally, with UTF-8 set. Then moved the website to 0.9.6.2 and it all went astray.

I had two backups : one made with PHPMyAdmin, and the other by MODx. None of them two worked.

I have removed UTF-8 from $database_connection_charset and it now works.

What I did was a fresh install of MODx instead of an upgrade and the $database_connection_charset parameter was set to UTF-8. In the original config.inc.php, it was set to ’’.
My two backup files were made with settings : $database_connection_charset=’’

UTF-8 is wrong -- it should be utf8. And if it was blank before, you were potentially inserting data with the wrong connection collation, depending on what your server defaults are.

Hmmm ... could this be another manager login page warning flag/check?