Import HTML not recognizing UTF-8 characters

I just imported a bunch of HTML files into MODX using the Import feature, and all the special characters get replaced with a question mark. The database is UTF-8, the system settings have UTF-8 for character encoding, and all the HTML files have UTF-8 for the meta content-type. What gives???

Not only that, I just preprocessed the HTML file, converting the special characters to HTML entities, and the same thing happens!!

Revolution 2.4.2 [ed. note: rainbowtiger last edited this post 10 years, 8 months ago.]

I wonder what PHP functions the import feature uses? There are several string functions in PHP that don't do utf-8.

Quote from: rainbowtiger at Jan 31, 2016, 07:32 PM

I just imported a bunch of HTML files into MODX using the Import feature, and all the special characters get replaced with a question mark. The database is UTF-8, the system settings have UTF-8 for character encoding, and all the HTML files have UTF-8 for the meta content-type. What gives???

Not only that, I just preprocessed the HTML file, converting the special characters to HTML entities, and the same thing happens!!

Revolution 2.4.2

Hi,

I once had the same issues, using Drupal import.

I could fix it this way:

1. Install this free editor: https://notepad-plus-plus.org/

2. Then open your html text, select all, and choose in Notepad+ : "CHARACTER > CONVERT TO UTF8 (WITHOUT BOM)

3. Save

Now "weird" characters will be corrected, this worked like a charm for me

Yeah, you should check what is the charset of the file. Lots of text editor has not UTF-8 as default charset and convert automatically the file into an iso charset or something else when you save it. The meta content-type is only used by the server/browser. It's different from the charset used by your file itself.

I was also having problems with the import truncating the content at special characters. It turned out to be malformed HTML entities in the text - no trailing semi-colon (?? apparently the original HTML pages were created using FrontPage). I had to hack the core/model/modx/import/modstaticimport.class.php and use some str_replace() functions to fix the offending text.

Perhaps not the greatest solution, but as it happens these were recipes so there were really only two entities causing the problem, the é in sauté and oven temperatures like 400°F. Plus this is just a temporary requirement for getting several thousand recipes imported, so it won't matter if it is overwritten on the next upgrade.

I was given a regular expression, but haven't tried it yet, at line 111 replace with

And maybe this line after it
[ed. note: sottwell last edited this post 10 years, 7 months ago.]