Weirdest problem of @-characters converting field to hex values when using snippet inside script tag

16 Posts

konservin Reply #1, 6 years, 7 months ago

I'm using a snippet to get values from a database (MariaDB) and assign them to js variables. Maybe there'd be a better way, but that's not my question.

It looks something like this:

<script type="text/javascript" charset="utf-8">
[[!activate]]
</script>

activate is a static file (DIRECTORY/activation.php), relevant part of which looks smth like this:

<?php
$hid = $_GET['hid'];
list ($app_id, $token, $mac) = explode(':', $hid);
$query = "SELECT * FROM AK_activations WHERE token = '$token'";
$userdata = $sql->Query($query);
var_dump($userdata);

(var_dump is, of course, for debugging)
Outcome is as it should:

["admin_name"]=> string(2) "EB"
["order_id"]=>string(2) "55"

etc.

But now the weirdest part: while all the fields are displayed as they should, they get converted to hex code whenever there's @ sign in the field. For example:
[email protected] will become different sequence of hex characters after each refresh:

["email"]=> string(15) "e& #114;& #x76;& #105;& #110;& #64;& #x70;& #117;& #x6e;& #x6b;& #x65;r& #x2e;& #x65;& #x65;"

or

["email"]=> string(15) "& #x65;r& #118;& #105;& #110;& #64;& #112;& #x75;& #x6e;& #x6b;& #101;& #x72;.& #x65;& #x65;"

or

["email"]=> string(15) "& #101;& #114;& #118;& #x69;& #110;& #64;& #x70;& #117;& #x6e;& #x6b;& #101;& #x72;& #46;& #101;e"

(I added a space after &, otherwise it'd have looked correct in this post that converts hex characters to utf8.)
while ervin @punker.ee (note the space before @) is correct:

["email"]=> string(16) "ervin @punker.ee"

Few further remarks:
1) It doesn't have to be valid e-mail to get hexed ("ervin@punker get's hexed, too)
2) it's nought to do with SQL field structure (VARCHAR 100, utf8_estonian_ci), as the other fields with @-sign inserted behave the same way,
3) I can't reproduce it with any other character (tried ≠,∆,∑,◊ and some others)
4) All is well when the snippet is outside of the <script> tag like this:

[[!activate]]<script type="text/javascript" charset="utf-8"></script>

5) I can't reproduce it outside MODX (see the correct-output test source outside MODX at http://test.myyt.ee/test.php?hid=55:5m5kkKIn0aJz/irI0gyZiA==:765ed33103c6e3cb08de92bb8e547008d6a5dbaed83e33b15a33b31c2ca44ddc, as opposed to not-correct-output source at https://www.myyt.ee/aktiveeri.html?hid=55:5m5kkKIn0aJz/irI0gyZiA==:765ed33103c6e3cb08de92bb8e547008d6a5dbaed83e33b15a33b31c2ca44ddc so it seems to have something to do with MODX Revo.

Tried googling it, but nothing comes up.. Any ideas?

This question has been answered by BobRay. See the first response.

[ed. note: konservin last edited this post 6 years, 7 months ago.]

24,544 Posts

BobRay Reply #2, 6 years, 7 months ago

I'm not sure what's causing that, but have a couple of thoughts:

$query("SET NAMES 'utf8'");

Also, it looks like the string is being run through json_encode() at some point, if you can escape the @ signs with backslashes before that happens it might solve the problem.

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

16 Posts

konservin Reply #3, 6 years, 7 months ago

Thank You BobRay,

Names are all set, every unicode character comes through, so that's not the cause.

Yes, I thought of escaping the @-sign upon populating the field, or even replacing it with triple-∆∆∆ and then re-replacing it, but it seems rather hacky..

Funnily, this doesn't work either. I changed the field value to ervin\@punker.ee and added few lines like this:

$query = "SELECT * FROM AK_activations WHERE token = '$token'";
$userdata = $sql->Query($query);
$userdata = $userdata[0];
var_dump($userdata);
foreach ($userdata as $key => $value) {
  $userdata[$key] = stripslashes($value);
}
var_dump($userdata);

Result is ["email"]=>string(16) "ervin\@punker.ee" on the first dump, but same hexed thing on the other..

I wonder, why is strings inside snippets inside <script> tags json-encoded only when there's @-sign present.

16 Posts

konservin Reply #4, 6 years, 7 months ago

I found the culprit: ObfuscateEmail plugin for MODx. I suspected it already, removed the plugin (don't recall installing it anyway), but nothing changed, so I thought the problem was elsewhere.

But even after removing the plugin (and clearing the cache) I found this at /core/cache/includes/elements/modplugin/2.include.cache.php:

<?php
/**
 * ObfuscateEmail plugin for MODx.
 * By Aloysius Lim.
 *
/ .... /
function replaceEntities($matches)
{
    $address = html_entity_decode($matches[1]);
    $replaced = '';
    / .... /
}

$output = &$modx->resource->_output;
$output = preg_replace_callback(email_regex(), "replaceEntities", $output);
$output = preg_replace_callback('/(mailto:)/', "replaceEntities", $output);
return;

Removed last two lines, and hexes are gone, so problem solved.

But.. solved until clearing the cache.. -> ObfuscateEmail plugin for MODx is back again!
I understand that it's actually another question but still: how to remove the thing, or modify it? It's not listed under MODX add-ons, it's not in core/components or core/packages. Only occurrences of 'obfuscate' or 'replaceEntities' are in core/cache/, and after deleting the whole cache manually and then clearing the cache at MODX admin panel they reappear.. How to successfully remove it?

discuss.answer

24,544 Posts

BobRay Reply #5, 6 years, 7 months ago

Look in the modx_site_plugins table in the DB and see if the plugin is still there.

Are you (or your host) running another cache (e.g., varnish, memcached, apc, etc.)?

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

16 Posts

konservin Reply #6, 6 years, 7 months ago

Quote from: BobRay at Sep 18, 2017, 10:11 PM

Look in the modx_site_plugins table in the DB and see if the plugin is still there.

Thank You very much! It was there, and after removing it the problem is solved.

24,544 Posts

BobRay Reply #7, 6 years, 7 months ago

I'm glad I could help. Thanks for reporting back.

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

Answered Weirdest problem of @-characters converting field to hex values when using snippet inside script tag