We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
  • Well, I can't put this off any longer. As the topic says, I have 10,000 web users belonging to different groups that I have to import into Revo. These web users have extended attributes, courtesy of WebLoginPE. Now, I can brute-force export and import the users using phpMyAdmin and a lot of fancy search-and-replace in the sql dump files, but I would much rather learn a more elegant way of dealing with this through a proper Revo import script.

    As far as the MD5 passwords, I'll be installing the pbkdf2Convert plugin that will eventually convert all the passwords as people login; although I strongly suspect that the vast majority of these users are long gone.

    In any case, I do have to get all these user into Revo, one way or another. Any advice or suggestions?
      Studying MODX in the desert - http://sottwell.com
      Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
      Join the Slack Community - http://modx.org
    • Well, I've solved everything except the extended attributes business. Now, interestingly enough, phpMyAdmin has a feature for exporting to JSON. So I imported the whole table into my local Revo installation, where I already have all the users imported (well, almost - I got the email in the wrong field, but that will be easy enough sort out)

      Now what I need to do is explode the resulting JSON export to get an array of the individual sets of fields. Then the hard part - I need to figure out is how to extract the internalKey field from the JSON so I can insert it into the appropriate user's "extended" field.
        Studying MODX in the desert - http://sottwell.com
        Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
        Join the Slack Community - http://modx.org
        • 3749
        • 24,544 Posts
        That's an ugly job, and trying to come up with an elegant solution will be really interesting. wink

        I'm doing something similar for a client coming from a non-MODX CMS. I think I'm going to export to a CSV file and then process that for the import.

        Where and how are the extended attributes stored (are they JSON strings)? I'd be tempted to process them separately (as long as the usernames are unique or the users will get the same ID in the new site).

        BTW, have you looked at Provisioner for this? I read that it's working again with the current versions.

          Did I help you? Buy me a beer
          Get my Book: MODX:The Official Guide
          MODX info for everyone: http://bobsguides.com/modx.html
          My MODX Extras
          Bob's Guides is now hosted at A2 MODX Hosting
        • It's a custom table; WebLoginPE uses a custom table and custom fields.

          I installed the same version of Evo on localhost, exported all of the web user and web group tables from the Evo site and imported them to localhost. I didn't want to be playing around on the live site!

          Then I wrote a quick and really dirty snippet to get the users for a group, and proceeded from there, working with each group of users. Some judicious copy/pasting and search-and-replace of the resulting user data gave me something I could import into the Revo user tables.

          A snippet in Revo involving getCollection (the abbreviated pdoTools version, the MODx version kept running out of memory) and joinGroup finally got all the users into their groups.

          So all that I have left to do is to get that JSON dump of the WebLoginPE extended table broken up and inserted. But it's 3:30 here, and I really need to shut down and get some sleep.
            Studying MODX in the desert - http://sottwell.com
            Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
            Join the Slack Community - http://modx.org
            • 3749
            • 24,544 Posts
            Remember that $profile->set('extended', $attributes) takes a regular PHP array, not JSON. The set() method converts it to JSON.

            Also, when you're doing this on a lot of users, I'd recommend using getIterator() instead of getCollection(). The args are identical but it will be much more memory-efficient.
              Did I help you? Buy me a beer
              Get my Book: MODX:The Official Guide
              MODX info for everyone: http://bobsguides.com/modx.html
              My MODX Extras
              Bob's Guides is now hosted at A2 MODX Hosting
            • See, I knew you'd know of a better way! That's why I did this on local, so I could play with it until I got it right. Besides, I have phpMyAdmin on local, and it's easier than command-line via SSH. Once I get it right, I can simply dump the local tables and import them to the remote via SSH command line.
                Studying MODX in the desert - http://sottwell.com
                Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
                Join the Slack Community - http://modx.org
              • Ok. Here's how I got the Evo WebLoginPD user_attributes_extended table converted into JSON strings and loaded in to the Revo user_attributes table's extended field.


                1. Use phpMyAdmin to get a JSON dump of the user_attributes_extended table.
                2. Edit the .json file and delete the leading comments and the enclosing [ ] tags.
                3. Load it up and process it in a snippet - ugly, and I had to increase my max_execution_time for it, but it worked.

                $file = 'assets/files/modx_user_attributes_extended.json';
                if(is_file($file)) {
                    $json = file_get_contents($file);
                    $array = explode('},',$json);
                    foreach($array as $key=>$value) {
                        $value = $value . '}'; // replace the closing } for the JSON rows
                        $row = json_decode($value); // get objects
                        $userId = $row->internalKey;
                        $user = $modx->getObject('modUser',$userId);
                        if (!$user) continue;
                        $profile = $user->getOne('Profile');
                        if (!$profile) continue;
                        $fields = $profile->get('extended');
                        $fields['value1'] = $row->value1;
                        $fields['value2'] = $row->value2;
                        ...
                        $profile->set('extended',$fields);
                        $profile->save();	
                    }
                }
                return;
                

                So 10,328 users imported. Now that I know how, it'll only take a few minutes to do it again for the dev site just before we take it live.
                  Studying MODX in the desert - http://sottwell.com
                  Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
                  Join the Slack Community - http://modx.org
                  • 10165
                  • 129 Posts
                  Thanks for posting this - I've got to attempt the exact same thing (users handled in webloginpe) so will give it a go - cheers


                  Quote from: sottwell at Oct 05, 2013, 02:01 PM
                  Ok. Here's how I got the Evo WebLoginPD user_attributes_extended table converted into JSON strings and loaded in to the Revo user_attributes table's extended field.


                  1. Use phpMyAdmin to get a JSON dump of the user_attributes_extended table.
                  2. Edit the .json file and delete the leading comments and the enclosing [ ] tags.
                  3. Load it up and process it in a snippet - ugly, and I had to increase my max_execution_time for it, but it worked.

                  $file = 'assets/files/modx_user_attributes_extended.json';
                  if(is_file($file)) {
                      $json = file_get_contents($file);
                      $array = explode('},',$json);
                      foreach($array as $key=>$value) {
                          $value = $value . '}'; // replace the closing } for the JSON rows
                          $row = json_decode($value); // get objects
                          $userId = $row->internalKey;
                          $user = $modx->getObject('modUser',$userId);
                          if (!$user) continue;
                          $profile = $user->getOne('Profile');
                          if (!$profile) continue;
                          $fields = $profile->get('extended');
                          $fields['value1'] = $row->value1;
                          $fields['value2'] = $row->value2;
                          ...
                          $profile->set('extended',$fields);
                          $profile->save();	
                      }
                  }
                  return;
                  

                  So 10,328 users imported. Now that I know how, it'll only take a few minutes to do it again for the dev site just before we take it live.