special characters getting lost and changed into weird characters

Hi, I am struggling with the following issue all day now. Hopefully somebody can help.

I have setup a MIGX which holds image and text data for a multi-column page layout. Everytime when there are special characters in the text, the first time when I save the page, all is fine. When viewing the page front-end, all special characters appear as they should.

However, when I edit the entered data, so right-mouse click on the grid > edit, the text data is changed, namely all special characters are lost and changed into weird characters. For example, any word that was put between (probably curled) ' ', now is put between ‘ and ’. Any ë in the text is changed into ë. Also spaces (particularly at end of sentence) seem to get converted to Â

The text data in the MIGX are all set as richtext (inputTVtype":"richtext") and edited from the tinyMCE editor that gets loaded in the MIGX form.

In comparison: When I enter text in a default content field of a page, also containing special characters, everything runs fine. After saving and re-editing the page, nothing changed. Here same again: editing is done via tinyMCE. Only change I see is that html entities get encoded, when I look at the raw html in tinMCE. But quotes or any special characters like ë or é just remain untouched as it should be.

My database is set at UTF-8 with collation utf8_general_ci; So I dont think this is a mismatch of charset problem.

It could just be caused by pasting from Word, which uses curly quotes.

Just to make sure, though, look at the DB in PhpMyAdmin and check the character set/collation for each table and text field. They don't always match the character set of the DB or MODX. Also, check the character set in core/config/config.inc.php.

If there's a mismatch, see this before trying to fix it: http://bobsguides.com/convert-db-utf8.html

richtext-editors are malicious monsters

Drop the text into notepad++ or some other neutral editor first, then copy it from there to the site.

My wife loves RTE like TIny etc. I hate them with a passion. Give me a napkin instead - at least it will keep it the way I wrote it.

Quote from: BobRay at Jun 04, 2013, 11:15 PM

Just to make sure, though, look at the DB in PhpMyAdmin and check the character set/collation for each table and text field. They don't always match the character set of the DB or MODX. Also, check the character set in core/config/config.inc.php.

In config.inc.php it reads: $database_connection_charset = 'utf8';
All tables in the DB are set to: utf8_general_ci

Its not a pasting text from Word problem. I tried all options: via notepad, and with no pasting at all: just entering texts with special characters. The first time after save everything is ok. Then when edit again via MIGX, the text gets changed. [ed. note: mick-nl last edited this post 13 years, 4 months ago.]

Quote from: Bruno17 at Jun 04, 2013, 11:20 PM

richtext-editors are malicious monsters

I tend to agree, but the point is I need some text editor for those textfields that my client can work with later (to do some updating on the info). BTW: I do nothing very strange - the whole thing I am creating is exactly what is described at: http://rtfm.modx.com/display/ADDON/MIGX.Varying+layout-boxes

Let me rephrase the question:
How come that when I enter text in a MIGX tv-textfield, the characters are changed AFTER I again edit it? While when I enter the very same text in a default content box (also richtext), the text hasnt changed when I again edit it.

Remember: the first time I enter the text and just do save page, everything is normal. The problem arises when I again want to edit the text via the MIGX interface.

could you post an example-text?

Quote from: Bruno17 at Jun 05, 2013, 05:26 AM

could you post an example-text?

Sure, [it's in dutch (hence the need for special characters, quite often)]
a short example of a text:

Zintuiglijke prikkeling maakt permanent deel uit van het leven; geuren van etherische oliën bij baden en voetmassages, vanuit de keuken waar gekookt wordt, het gestreken wasgoed, het brandende hout in de kachel, de zelf geplukte kruiden uit de tuin.

after edit again this has changed to:

Zintuiglijke prikkeling maakt permanent deel uit van het leven; geuren van etherische oliën bij baden en voetmassages, vanuit de keuken waar gekookt wordt, het gestreken wasgoed, het brandende hout in de kachel, de zelf geplukte kruiden uit de tuin.

Hm... wonder if it's something to do with the javascript reading the JSON from the database?

did some tests saving, editing, reading, html-view-editing, switched between different formtabs, all of that several times with your text, but could not reproduce it. The text remains still the same as the initial input.

revo 2.2.7/modx-cloud/latest tinyMCE/latest MIGX/firefox 21.0 with multiple formtabs and one tinyMCE - field