We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 10378
    • 375 Posts
    Hi,

    I currently have a big performance problem with querying the MODX users table and fetching all user ID's + collecting them in an array.

    Fetching about 4500 users from the database - the following code part has a run time of about 20 seconds with the getIterator method and about 30 seconds when using getCollection.

    $c = $this->modx->newQuery('modUser');
    $c->leftJoin('modUserProfile', 'Profile');
    $c->leftJoin('GoodNewsGroupMember', 'GroupMember', 'GroupMember.member_id = modUser.id');
    $c->leftJoin('GoodNewsCategoryMember', 'CategoryMember', 'CategoryMember.member_id = modUser.id');
    $c->where(array(
        'modUser.active' => true,
        'Profile.blocked' => false,
        'GroupMember.goodnewsgroup_id:IN' => $groups,
    ));
    $c->orCondition(array(
        'CategoryMember.goodnewscategory_id:IN' => $categories,
    ));
    $c->select($this->modx->getSelectColumns('modUser', 'modUser', '', array('id')));
    //$c->select($this->modx->getSelectColumns('modUserProfile', 'Profile', '', array('blocked')));
    
    //$users = $this->modx->getCollection('modUser', $c);
    $users = $this->modx->getIterator('modUser', $c);
    
    foreach ($users as $user) {
        $userID = $user->get('id');
        $recipients[$userID] = array(self::GON_USER_NOT_YET_SENT,0);
    }
    


    Any hints on how to improve the performance of this code part?

    Thanks,
    Martin

    This question has been answered by gadgetto. See the first response.

      Freelancer @bitego http://www.bitego.com
      ---
      GoodNews - one of the most advanced and integrated Group Mailer premium add-ons for MODX Revolution!
      More infos here: http://www.bitego.com/extras/goodnews/
      • 22303 MODX Staff
      • 10,725 Posts
      Quote from: gadgetto at Jan 10, 2014, 02:56 PM

      Any hints on how to improve the performance of this code part?
      Without knowing the details of your custom tables, my first inclination is to say, don't try to grab that many records at once. My second is to make sure the fields being used in your joins/where conditions are properly indexed. Finally, you don't want to load all those objects. If you ultimately want just an array of id's, you should only select that column in your query and then use PDOStatement->fetchAll(PDO::FETCH_COLUMN) instead of any xPDO methods that load the data into objects.
        • 10378
        • 375 Posts
        Quote from: opengeek at Jan 10, 2014, 03:05 PM

        Without knowing the details of your custom tables, my first inclination is to say, don't try to grab that many records at once.

        As I need to build a single array with all user IDs matching the query, I'm afraid I need to grab all records at once.

        Quote from: opengeek at Jan 10, 2014, 03:05 PM

        My second is to make sure the fields being used in your joins/where conditions are properly indexed.

        The necessary fields are indexed properly (I think):

            <object class="GoodNewsGroupMember" table="goodnews_group_members" extends="xPDOSimpleObject">
                <field key="goodnewsgroup_id" dbtype="int" precision="10" attributes="unsigned" phptype="integer" null="false" default="0" index="index" />
                <field key="member_id" dbtype="int" precision="10" attributes="unsigned" phptype="integer" null="false" default="0" index="index" />
        
                <index alias="goodnewsgroup_id" name="goodnewsgroup_id" primary="false" unique="false" type="BTREE">
                    <column key="goodnewsgroup_id" length="" collation="A" null="false" />
                </index>
                <index alias="member_id" name="member_id" primary="false" unique="false" type="BTREE">
                    <column key="member_id" length="" collation="A" null="false" />
                </index>
        
                <aggregate alias="GoodNewsGroup" class="GoodNewsGroup" local="goodnewsgroup_id" foreign="id" cardinality="one" owner="foreign" />
                <aggregate alias="User" class="modUser" local="member_id" foreign="id" cardinality="one" owner="foreign" />
            </object>
        


        Quote from: opengeek at Jan 10, 2014, 03:05 PM

        Finally, you don't want to load all those objects. If you ultimately want just an array of id's, you should only select that column in your query and then use PDOStatement->fetchAll(PDO::FETCH_COLUMN) instead of any xPDO methods that load the data into objects.

        I'm not yet familiar with this method but will try this (hope this is in the RTFM) - thanks opengeek!
          Freelancer @bitego http://www.bitego.com
          ---
          GoodNews - one of the most advanced and integrated Group Mailer premium add-ons for MODX Revolution!
          More infos here: http://www.bitego.com/extras/goodnews/
          • 3749
          • 24,544 Posts
          Do you really want blocked and inactive users in the array? It appears that you'll get them with the or condition. Maybe I'm missing something.


          Also, is it possible that you could get all the user IDs you need from the GoodNewsCategoryMember table without any joins at all? It's not clear what you're actually with the user information so maybe that doesn't make any sense.
            Did I help you? Buy me a beer
            Get my Book: MODX:The Official Guide
            MODX info for everyone: http://bobsguides.com/modx.html
            My MODX Extras
            Bob's Guides is now hosted at A2 MODX Hosting
            • 28042 ☆ A M B ☆
            • 24,524 Posts
            The pdoTools library is pretty fast. You might find it interesting to see how the pdoUsers snippet does it.
              Studying MODX in the desert - http://sottwell.com
              Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
              Join the Slack Community - http://modx.org
              • 10378
              • 375 Posts
              Quote from: BobRay at Jan 10, 2014, 06:18 PM
              Do you really want blocked and inactive users in the array? It appears that you'll get them with the or condition. Maybe I'm missing something.

              Hi Bob,

              really?

              $c->where(array(
                  'modUser.active' => true,
                  'Profile.blocked' => false,
                  'GroupMember.goodnewsgroup_id:IN' => $groups,
              ));
              $c->orCondition(array(
                  'CategoryMember.goodnewscategory_id:IN' => $categories,
              ));
              


              Does the or condition exclude all other conditions?
                Freelancer @bitego http://www.bitego.com
                ---
                GoodNews - one of the most advanced and integrated Group Mailer premium add-ons for MODX Revolution!
                More infos here: http://www.bitego.com/extras/goodnews/
                • 10378
                • 375 Posts
                Quote from: sottwell at Jan 11, 2014, 02:16 AM
                The pdoTools library is pretty fast. You might find it interesting to see how the pdoUsers snippet does it.

                Thanks Susan, I'll have a look at this!
                  Freelancer @bitego http://www.bitego.com
                  ---
                  GoodNews - one of the most advanced and integrated Group Mailer premium add-ons for MODX Revolution!
                  More infos here: http://www.bitego.com/extras/goodnews/
                  • 3749
                  • 24,544 Posts
                  I'm not a whiz at xPDO queries, but you're basically saying, "Give me the records where these three conditions are true OR this other condition is true."

                  Check the SQL and look at the parentheses in the WHERE clause:

                  $c->prepare();
                  echo $c->toSQL();



                  I think what you want is probably something like this:


                  WHERE
                    modUser.Active => true,
                    AND
                    Profile.blocked => false
                    AND (
                        'GroupMember.goodnewsgroup_id:IN' => $groups,
                        OR
                        'CategoryMember.goodnewscategory_id:IN' => $categories,
                    )



                  I'm not sure about the optimum way to express that in xPDO.
                    Did I help you? Buy me a beer
                    Get my Book: MODX:The Official Guide
                    MODX info for everyone: http://bobsguides.com/modx.html
                    My MODX Extras
                    Bob's Guides is now hosted at A2 MODX Hosting
                  • discuss.answer
                    • 10378
                    • 375 Posts
                    OK - I implemented opengeek's hint and execution time drops from 21,1 seconds to 0,1 seconds! This is way cool!

                    Here is the new code part:

                            // Select subscribers based on groups/categories
                            $tblUsers                  = $this->modx->getTableName('modUser');
                            $tblUserAttributes         = $this->modx->getTableName('modUserProfile');
                            $tblGoodNewsGroupMember    = $this->modx->getTableName('GoodNewsGroupMember');
                            $tblGoodNewsCategoryMember = $this->modx->getTableName('GoodNewsCategoryMember');
                            
                            $groupslist = implode(',', $groups);
                            $categorieslist = implode(',', $categories);
                            
                            $sql = "SELECT DISTINCT {$tblUsers}.id
                                    FROM {$tblUsers} 
                                    LEFT JOIN {$tblUserAttributes} ON {$tblUserAttributes}.internalKey = {$tblUsers}.id
                                    LEFT JOIN {$tblGoodNewsGroupMember} ON {$tblGoodNewsGroupMember}.member_id = {$tblUsers}.id
                                    LEFT JOIN {$tblGoodNewsCategoryMember} ON {$tblGoodNewsCategoryMember}.member_id = {$tblUsers}.id
                                    WHERE ({$tblGoodNewsGroupMember}.goodnewsgroup_id IN ({$groupslist}) OR {$tblGoodNewsCategoryMember}.goodnewscategory_id IN ({$categorieslist}))
                                    AND {$tblUsers}.active = 1 
                                    AND {$tblUserAttributes}.blocked = 0";
                    
                            $query = $this->modx->query($sql);
                            if ($query) {
                                $users = $query->fetchAll(PDO::FETCH_COLUMN);
                            }
                    
                            // Initialize each userid with status not_yet_sent + timestamp placeholder 0
                            foreach ($users as $id) {
                                $recipients[$id] = array(self::GON_USER_NOT_YET_SENT,0);
                            }
                    
                    


                    I'm not very good in writing effective SQL queries! It probably could be solved in better ways - but it now works as I expected!
                      Freelancer @bitego http://www.bitego.com
                      ---
                      GoodNews - one of the most advanced and integrated Group Mailer premium add-ons for MODX Revolution!
                      More infos here: http://www.bitego.com/extras/goodnews/
                      • 10378
                      • 375 Posts
                      Quote from: BobRay at Jan 11, 2014, 02:23 PM
                      I'm not a whiz at xPDO queries, but you're basically saying, "Give me the records where these three conditions are true OR this other condition is true."

                      Check the SQL and look at the parentheses in the WHERE clause:

                      You are right Bob, the query looks wrong. The strange part is, that it works and returns the right selection of users.
                      But I now solved it in another way (see my last post) and it works like a charm!
                        Freelancer @bitego http://www.bitego.com
                        ---
                        GoodNews - one of the most advanced and integrated Group Mailer premium add-ons for MODX Revolution!
                        More infos here: http://www.bitego.com/extras/goodnews/