We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 727
    • 502 Posts
    I’ve been struggling on and off with this problem for several weeks. I’ve now managed to produce a simple situation where the problem is reproduced. It only seems to happen with MODx.

    Problem: Accented characters don’t seem to work properly in snippets

    Severity: This is critical for development of non-english web sites

    How to Reproduce:

    1. Create a MySQL table called "test" with the following fields:

    - id, int(11), unsigned, auto increment, primary
    - name, text(200) (utf8)

    2. Create a php file called "test.php":

    <?php
    function tok($string) {
      $want = "Öffentlich";
      return "(".$want.")";
    }
    ?>


    3. Create a snippet called Insert:

    <?php
    require_once "test.php";
    $dbTable = array();
    $dbTable['name'] = tok('foo');
    $modx->db->insert($dbTable, 'test');
    ?>


    4. Create a new document with just a call to the Insert snippet.

    Load the page and look at the contents of the name field in the table. You will see:

    (


    Things to try:

    1. Try editing the tok() function to use strcmp on the variable $want. It won’t match "Öffentlich". Weird.

    2. Move the tok function into the snippet. Now it works!

    I’m using 0.9.5 RC3, Linux (Gentoo 2006.1), PHP 5.1.6, MySQL 5.0.26.

    I really hope someone can give me some clues or things to try before I pull my remaining hair out. I need to get this German based website done. :’(

    Andy
      • 22303 MODX Staff
      • 10,725 Posts
      [*] What is the character set and collation sequence of your database, and for the MySQL server? You might want to read up on this thread...
      [*] 0.9.5 RC3? But why?
      [*] 0.9.6 (RC1 and current SVN trunk, marked as RC2) address the issues discussed in the above thread.
        • 727
        • 502 Posts
        Thanks for replying. I think this is a PHP problem, not a MySQL problem. I am using MySQL to get debug output essentially. Here is why:

        If you move the tok() function from the included file test.php into the snippet itself, everything works. By that I mean that the following is inserted into the MySQL table:

        (Öffentlich)


        This rules out a problem with MySQL. Same PHP code, just moving a function!? And it works? Why?
        Also try entering a strcmp inside the tok function. I can’t get it to work because I think the value of $want is mangled for some reason.

        I would be hugely grateful if someone would give my steps a try and see if it works on their setup. Then we can look at the differences.

        Some additional information: I don’t seem to have any problems displaying accented characters in MODx pages or the manager. Also I seem to be able to insert data in the MySQL database containing accented characters just fine. It only seems to fail in this specific situation.

        To answer your questions:

        When I started working on this site 0.9.5 RC3 was the latest release. I’ve been too busy trying to solve this weird problem to upgrade.

        MySQL charset is UTF-8 Unicode (utf8)
        MySQL connection collation is utf8_unicode_ci
        Collation of all database tables is utf8_general_ci

        Andy
          • 727
          • 502 Posts
          Update: I upgraded my site to 0.9.6 RC1 and the problem still exists and is the same.

          Note that if I move the snippet code to a php file and call it (for example, test2.php) then the PHP code executes fine. So MODx is involved somehow.

          Andy
            • 727
            • 502 Posts
            Has anyone been able to recreate this problem? sad

            Andy
              • 4626
              • 57 Posts
              Just a wild guess...

              Could there be a problem of different document encodings between the included file and your MODx page?
                • 727
                • 502 Posts
                I can show accented characters just fine in MODx pages. It’s when I use strcmp() I run into a problem. To see what I was comparing with I output the variables to a MySQL table and I see one of the PHP variables is mangled in some way.

                How do you set the encoding of a plain PHP file?

                Thanks for replying! smiley

                Andy
                  • 4626
                  • 57 Posts
                  The encoding of the file you set in your texteditor – but in my test I don’t get the same results...

                  Are you sure you are using strcmp the right way? It returns an int < 0 if str1 < str2, int > 0 if str > str2 and 0 if they are equal.
                    • 33372
                    • 1,611 Posts
                    This looks like a Magic Quotes or escaped character bug in PHP more than a MODx issue to me.

                    Why are you using the HTML entity in your $want string? Why not "Öffentlich"? And why are you using double quotes if there are no variables to parse in that string? I would alter your quotes (use single wherever possible) and test different character encodings. I also try to avoid returning anything but a precalculated value, since I’ve had some odd results otherwise. So maybe $output=’(’.$want.’)’; followed by return $output; would eliminate that possibility. What happens if you output tok(’foo’); to the screen instead of inserting it into the database (is it correct at that point)?

                    I know that this is just sample code, but man is it wacky!
                      "Things are not what they appear to be; nor are they otherwise." - Buddha

                      "Well, gee, Buddha - that wasn&#39;t very helpful..." - ZAP

                      Useful MODx links: documentation | wiki | forum guidelines | bugs & requests | info you should include with your post | commercial support options
                      • 727
                      • 502 Posts
                      Hi Zap, the HTML entity is a product of this forum software. When I made that post and read it afterwards the umlaut-O appeared correctly. Now for some reason it isn’t. So I think that is an SMF issue. If you can tell me how to fix it that would be great. smiley

                      I think it is connected to MODx somehow because if I move the snippet code to a simple PHP file and access the PHP file from my browser, it works perfectly.

                      I’m using double quotes out of habit (too much C programming) - poor I know, but it should still work.

                      How do I test different character encodings? I’m using nano as my editor. Accented characters appear ok in the PHP files and I can output them to web pages just fine. I can also store them in the MySQL database just fine. Does that indicate that the encodings are correct?

                      I can return a precalculated value, but this is just a simple test case for my real problem:

                      $want = 'Öffentlich';
                      if (!strcmp($string, $want)) {
                        return 'TOKEN1';
                      }
                      return 'TOKEN2';


                      When this code failed I started to look at what the problem was with strcmp. When I output $want, I saw it was mangled.

                      Andy