I apologise, since I should have checked the facts before pointing the finger at php’s inbuilt string functions for the truncation of the description field.
I wrote that post at 1.44am, which explains why I wasn’t being very coherent! I’ll describe the issue in more detail:
The truncation actually occurs at the point of entry into the mysql database, since the description column in the site_content table is limited to 255 bytes varchar(255).
Let me see if I can demonstrate the problem. First, here is a simple html form, with a text input field with maxlength="255" (just like the modx description field). This form posts to itself and displays the results.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Maxlength Form Text</title>
</head>
<body>
<form action="<?php echo htmlspecialchars($_SERVER['REQUEST_URI']); ?>" method="post">
<fieldset>
<input type="text" maxlength="255" name="text" />
<input type="submit" />
</fieldset>
</form>
<?php
if ( isset( $_POST['text'] ) )
{
echo '<p>"' . $_POST['text'] . '"</p>';
}
?>
</body>
</html>
You can type a maximum of 255 characters into this field, whatever the encoding. So, for example, here are 255 single byte characters:
012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234
Here are 255 multiple byte characters (these happen to be 3 bytes each in utf-8)...
０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４
and here is a mixed bunch of 255 single and multibyte characters.
かわいいasdasかわい23<>いかわいいかわいいasdasかわいいかわいいかわいいかわいいかssわいいかわいいかわいいかわいいかわいasdasdかわいいかわいいかわいいかasdasdわいいかわいいかわいいかわいいasdasかわい23<>いかわいいかわいい asdasかわいいかわいいかわいいかわいいかssわいいかわいいかわいいかわいいかわいasdasdかわいいかわいいかわいいかasdasdわいいかわいいかわいいkaかわいいasdasかわい23<>いかわいいかわいいasdasかわいいかわいいかわいいかわいいか
You can submit any of those strings in the html form and you get back what you put it. Now try the same with the MODx description field. Copy and past a string in, save the document, and then check that the string is correctly reproduced from end to end.
For me at least (modx 0.9.6.2, utf-8 everywhere), the single byte character string works fine.
The multibyte character is truncated at 255 bytes, which happens to mean that I get back exactly 85 of these 3-byte utf-8 characters:
０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４５６７８９０１２３４
When I try the mixed single byte multibyte string there is a problem. I get 115 characters back, the last of which is an incomplete and invalid utf-8 character:
かわいいasdasかわい23<>いかわいいかわいいasdasかわいいかわいいかわいいかわいいかssわいいかわいいかわいいかわいいかわいasdasdかわいいかわいいかわいいかasdasdわいいかわいいかわいい�
This occurs because the string has been truncated at 255 bytes, which, in this case, doesn’t lie on a character boundary.
It has happened several times now that a description has been written that is ~100 Japanese characters and, after saving, the description has been truncated generating invalid unicode. If, like I do, you serve your pages as application/xhtml+xml to browsers that support it, this error stops all parsing and display of the page by the browser (which is an argument against application/xhtml+xml, I know).
I know that this is going slightly off the original topic now, but would it be possible that, before modx inserts a string into a database field that has a fixed length in bytes, it could truncate it in a safe way using the
mb_substr function, if it is available that is? Doing this on every mysql call could potentially have performance issues, but only doing it when pages are submitted via the manager might be a good compromise.
Cheers.
MODX Staff
MODX Staff
