special characters getting lost and changed into weird characters

It sounds like the javascript is having problems parsing the JSON string on load. Handling utf-8 in JSON can be tricky.

Quote from: sottwell at Jun 06, 2013, 11:57 AM

It sounds like the javascript is having problems parsing the JSON string on load. Handling utf-8 in JSON can be tricky.

Hi, I have no idea what exactly happens technically when the MIGX is opened again. Is the JSON data being read from the actual database and parsed via javascript? Or is the text parsed from another (temporary?) source and then parsed via js?

One thing is very clear. After the text is entered first time, all characters appear as they should and when I look in the database, they exactly look the same as entered, so ë en é etc.

When I then look at the source of the manager page: these characters appear as encoded entities (I think its called that way), for ë thats \u00eb and for é \u00e9

To make it even more interesting: I tested a local windows installation of modx (xampp) and there I do not have this problem at all !! [ed. note: mick-nl last edited this post 13 years, 4 months ago.]

Is this using the rich text editor? Is the local installation using the same rte?

Quote from: sottwell at Jun 06, 2013, 06:27 PM

Is this using the rich text editor? Is the local installation using the same rte?

Yes, all the same, with tinyMCE. As I pointed out earlier, however, that editor isnt the culprit. It also happens when using plain html textarea for the text data.

Ah. Sorry, I missed that. Just trying to think of what it could be.

You might check the character set at the top when you do a "view source." Your template may be setting a wrong character set that's confusing the browser.

Quote from: BobRay at Jun 08, 2013, 02:47 AM

You might check the character set at the top when you do a "view source." Your template may be setting a wrong character set that's confusing the browser.

Both backend and frontend, when I view the source, it says: <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

It sounds like there are just some bad characters in the text, unless an individual field in the DB is set to some other character set. You might double-check the individual fields in the MIGX db table(s). They don't necessarily match the encoding of the table.

Quote from: mick-nl at Jun 06, 2013, 05:29 AM

It happens in both cases!
So after I close the MIGX form and save the input ('done' button), and then open the MIGX again, the characters have already changed.

Interestingly, in the grid table, the text looks normal, after I have entered it in the MIGX. However, as soon as the MIGX form is again opened, then it happens!

Quote from: mick-nl at Jun 06, 2013, 08:34 AM

Quote from: Bruno17 at Jun 06, 2013, 08:18 AM

seems to be client-side, then.
Did you try it on different browsers, different OSs, different machines?

Tried it with: Chrome, FF21, Safari 5.1.7, IE 10, yet all on Windows 7 and same machine.
Cleared cache from manager often and also cleared browser caches between tries.

Its really very strange isnt it?

Quote from: mick-nl at Jun 06, 2013, 12:22 PM

Quote from: sottwell at Jun 06, 2013, 11:57 AM

It sounds like the javascript is having problems parsing the JSON string on load. Handling utf-8 in JSON can be tricky.

Hi, I have no idea what exactly happens technically when the MIGX is opened again. Is the JSON data being read from the actual database and parsed via javascript? Or is the text parsed from another (temporary?) source and then parsed via js?

One thing is very clear. After the text is entered first time, all characters appear as they should and when I look in the database, they exactly look the same as entered, so ë en é etc.

When I then look at the source of the manager page: these characters appear as encoded entities (I think its called that way), for ë thats \u00eb and for é \u00e9

To make it even more interesting: I tested a local windows installation of modx (xampp) and there I do not have this problem at all !!

this all sounds very strange.

Thinking loud:

I thought it must happen client-side, somewhere between converting the the form-datas to json (and putting it into a hidden-field) and converting it from json (after reading it from our hidden-field) before filling the form, because you are saying it happens also when you just reopen the MIGX-form-window without saving the resource.

ok, before the form is created, the json is sended to the server to fill out the form.
this means it could also happen on server-side.

when I'm trying your text on a modx-cloud-installation this is working.
when you try it on a localhost-installation it is working too.
seems only with one specific installation on a specific server doesn't handle the json-converting stuff correctly.

so my conclusion for now: It must be a server-side issue with json-encoding/decoding
but has nothing todo with any db-stuff.
It must happen before anything is stored into the db, because of db-stuff isn't envolved when you just save the form (into the hidden MIGX-field) and rereading (sending to server and create the formdatas) it.

this may be the line where it happens:

https://github.com/Bruno17/MIGX/blob/master/core/components/migx/processors/mgr/fields.class.php#L71 [ed. note: Bruno17 last edited this post 13 years, 4 months ago.]

Thank you Bruno, for your comments and all work on this. Best guess, indeed, there must be something very specific to the server installation, not handling json correctly. Later I will try to contact my provider on this.

I have decided now, to pull out all text data fields from the MIGX and put them in separate resources (in the default content field, resource: non published, not in menu etc). Then in the MIGX I replaced the text fields with resource lists input TV, which point to the separate resources holding the texts. Using the "fastfield plugin" for the chunk code (making the front end page), I then have the resource ID's be replaced with the corresponding text resources. That all works flawlessly.