We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 31037
    • 358 Posts
    I’m trying to import 28.000 users from an old Mambo installation through a snippet I’ve made. (Not an easy task for a beginner, trying to convert Mambo users extended by ComProfiler to MODusers and MODProfiles with extended data as jSon... but great for learning).

    My problem is that the log shows errors (and the post isn’t saved):
    INSERT INTO `modx_users` (`id`, `username`, `password`, `cachepwd`, `class_key`, `active`) VALUES ('1922', 'Malmö', '', '', 'modUser', '1')
    Array
    (
        [0] => 23000
        [1] => 1062
        [2] => Duplicate entry 'Malmö' for key 'username'
    )

    There are for sure no duplicate usernames and I’ve spent a lot of hours trying to find the problem.

    It seems to be that when saving a user, $userObject->save(), it treats the user name "Göran" as the same as "Goran" and so on:
    Could not save user, skipping user and profile for: 5209 (lonn) [lönn exists]
    Could not save user, skipping user and profile for: 5469 (jönsson) [jonsson exists]
    User with no Comprofiler data detected, skipping: 6055
    Could not save user, skipping user and profile for: 9246 (asa) [Åsa exists]
    Could not save user, skipping user and profile for: 9393 (sötis) [Sotis exists]

    Any suggestions what the problem could be? As I can’t believe there is a bug in MODx saving user objects, it perhaps have to do with MySql or my enviroment? Or perhaps some nice switch somewhere that solves everyting?

    I tried to do a search but could not find a similar problem.

    I use Xampp 1.7.3 on windows, MySql 5.1.41, PHP 5.3.1, Revo 2.0.4-pl2. If needed I’ll post more enviroment data.

    Now I’ll start to try converting Mambo comments into MODx. Revo is very nice to work with, takes a while to understand at first but then everything is so easy and obvious!
      • 3749
      • 24,544 Posts
      I wonder if it might be a character set issue. Have you looked at the table itself and the individual fields to make sure the character set encoding matches the setting for the DB and for MODx?
        Did I help you? Buy me a beer
        Get my Book: MODX:The Official Guide
        MODX info for everyone: http://bobsguides.com/modx.html
        My MODX Extras
        Bob's Guides is now hosted at A2 MODX Hosting
        • 31037
        • 358 Posts
        Good suggestion, don’t think that is the problem, but I can use it as a workaround by converting ö to ö and so on.

        But tested this on another server with a clean install of Revo:
        $usr = $modx->newObject(modUser);
        $usr->set('username', 'Ake');
        $usr->save();
        
        $usr = $modx->newObject(modUser);
        $usr->set('username', 'Åke');
        $usr->save();
        
        $usr = $modx->newObject(modUser);
        $usr->set('username', 'Oken');
        $usr->save();
        
        $usr = $modx->newObject(modUser);
        $usr->set('username', 'Öken');
        $usr->save();

        Got me only two new users (Ake, Oken) and the error in the log:
        [2010-11-01 12:12:15] (ERROR @ /index.php) Error 23000 executing statement:
        INSERT INTO `modx_users` (`username`, `password`, `cachepwd`, `class_key`, `active`) VALUES ('Åke', '', '', 'modUser', '1')
        Array
        (
            [0] => 23000
            [1] => 1062
            [2] => Duplicate entry 'Åke' for key 2
        )
        
        [2010-11-01 12:12:15] (ERROR @ /index.php) Error 23000 executing statement:
        INSERT INTO `modx_users` (`username`, `password`, `cachepwd`, `class_key`, `active`) VALUES ('Öken', '', '', 'modUser', '1')
        Array
        (
            [0] => 23000
            [1] => 1062
            [2] => Duplicate entry 'Öken' for key 2
        )

        Is this a bug or just me missning something?
          • 22303 MODX Staff
          • 10,725 Posts
          Sounds like your database collation (and thus the collations of all text fields in your MODx tables) are wrong. Are they set to latin1_swedish_ci by chance?
            • 31037
            • 358 Posts
            Noop, but the table I’m importing from is latin1. But tested on another installtion too where:

            MODx config:
            $database_connection_charset = ’utf8’;

            DEFAULT_CHARACTER_SET_NAME
            utf8

            DEFAULT_COLLATION_NAME
            utf8_general_ci

            And on the created tables:
            Table COLLATION:
            utf8_general_ci

            I created a snippet with the following content:
              $usrObject = $modx->newObject('modUser');
              $usrObject->set('username', 'Asa');
              $usrObject->save();
              
              $usrObject = $modx->newObject('modUser');
              $usrObject->set('username', 'Åsa');
              $usrObject->save();
              
              $usrObject = $modx->newObject('modUser');
              $usrObject->set('username', 'Ake');
              $usrObject->save();
              
              $usrObject = $modx->newObject('modUser');
              $usrObject->set('username', 'Öken');
              $usrObject->save();

            Did not copy any text, wrote it directly in manager to be sure I didn’t pasted something stange.

            Got the errors in log as described above and Åsa isn’t created.

            If I have a user "Öken" in database and try to create "Oken" in manager I get "Username already in use".

            If I create in manager user "Jörgen" I cant’t create "Jorgen" in manager.

            EDIT:

            If I create directly in the table "Ösa" and "Osa" I get "Duplicate entry "Osa" for key 2"

            The error is certenly at database level. Not a MODx problem.

            Perhaps I shouldn’t store ÅÄÖ without encoding them in some way? But I’ve done that all the time before without any problem...

            Well, I’ll try Google it.

            Although not a MODx problem, perhaps someone could give me an advise. Otherwise, I guess I’ll just html encode it before storing the user names.

            EDIT#2:

            Will this not be a problem for many people when Revo gets more used? In Evo the "username" column in the table isn’t set to unique, it was the php code in the login snippet checking to be sure user name didn’t exist. (At least I think so.)
              • 22303 MODX Staff
              • 10,725 Posts
              You’ll have to use utf8_encode or possibly iconv to try and convert the strings from the latin1 table to valid utf8 strings then. You could also try using mysqldump with --default_character_set=utf8 on the table and reimporting it to see if MySQL itself can make the conversion appropriately itself.
                • 31037
                • 358 Posts
                As I said under my "Edit" above, this error comes from the "username" market as unique in the table:

                "If I create directly in the table "Ösa" and "Osa" I get "Duplicate entry "Osa" for key 2".

                Doesn’t matter where or how I enter it, in manager, through code or directly in the database.

                Latin1 has nothingh to do with it as the latest test I made was on an UTF8 database entering letters from my keyboard.

                I also tested to create a new table, in a new empty database, setting a text column to unique, can’t put "Asa" and "Åsa" without getting error, so this seems to be a "feature" in MySql.

                BUT: As long as "username" is set to unique in the table there will be problems for people entering users in manager.

                I’ll solve it by converting to html entities, but that doesn’t help end users trying to register "Asa" and "Åsa" (bad example) in manager or front end.

                Perhaps someone else can try adding two users Asa and Åsa in the manager to see if it works?

                  • 31037
                  • 358 Posts
                  Problem solved... kind of...

                  Ok, this is how it works:

                  In MySql utf-general-ci letters like å/a are treated as same letter when checking if is unique (username is defined as unique in the user table).

                  Solution is to set it to some more specific than general-ci, for me swedish-ci works, now I can enter Asa and Åsa as users.

                  Most people use general-ci and could end up having problem with international letters.

                  Don’t think having unique index on "username" is an good idea, but for me it works now with changes above.

                  So many wasted hours looking for non existing errors in my code lol...

                    • 3749
                    • 24,544 Posts
                    Interesting. Thanks for reporting back.

                    Did you try utf8_unicode_ci?

                      Did I help you? Buy me a beer
                      Get my Book: MODX:The Official Guide
                      MODX info for everyone: http://bobsguides.com/modx.html
                      My MODX Extras
                      Bob's Guides is now hosted at A2 MODX Hosting
                      • 31037
                      • 358 Posts
                      Quote from: BobRay at Nov 01, 2010, 05:01 PM
                      Did you try utf8_unicode_ci?
                      Yes, utf8_unicode_ci show same result as utf8_general_ci, that means not working. I had to set it to utf8_swedish_ci.
                      There might be a problem in the future when people start to convert users from Evo to Revo, for non english languages at least. Or it might not. smiley