We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 38954
    • 24 Posts
    i write cusstom snippet which import data from .csv file to mysql database using xPDO model.
    it works fine when i load less than 50 thousand rows, but once i want load all data from csv (about 80 thousand rows) something strange happens. i look in phpMyAdmin when snippet works and once it reach (about) 50 thousand rows he start to work again from first row (i can see the number of rows in phpmyadmin). Probably my snippet just start again themselev (i don't know why), because i start from deleting table and make it again. Here's the code:
    <?php
    $xpdo = new xPDO($dsn,'*****','*****');
    echo $o=($xpdo->connect()) ? 'Connected' : 'Not Connected';
    $core = $modx->getOption('core_path');
    $xpdo->addPackage('csvcars', $core.'model/');
    $manager = $xpdo->getManager();
    $manager->removeObjectContainer('csvCar') ? 'Removed' : 'Not removed';
    $manager->createObjectContainer('csvCar') ? 'Created' : 'Not created';
    unset($manager);
    set_time_limit(600);
    $fp = fopen("data.csv", "r");
    $k = 0; 
    while (($fields = fgetcsv($fp, 0, ',')) !== false) { 
        $myCars = $xpdo->newObject('csvCar');
        $myCars->fromArray(array(
            'lot' => $fields[7],
            'vin' => $fields[21],
            'image' => $fields[40],
            'saledate' => $fields[2],
            'year' => $fields[9],
            'make' => $fields[10],
            'model' => $fields[12],
            'place' => $fields[35],
            'odometer' => $fields[22],
            'bid' => $fields[33],
            'damage' => $fields[15],
            'repair' => $fields[25],
            'bodystyle' => $fields[13],
            'color' => $fields[14],
            'engine' => $fields[26],
            'drive' => $fields[27],
            'fuel' => $fields[29],
            'cylinders' => $fields[30],
            'acv' => $fields[24],
            'secdamage' => $fields[16],
            'keys' => $fields[19],
            'note' => $fields[34],
            'facility' => $fields[1],
            'saletime' => $fields[4],
            'salezone' => $fields[5],
            'salestatus' => $fields[32],
            'titlestate' => $fields[17],
            'titletype' => $fields[18],
        ));
        $myCars->save();
        unset($myCars);
    }
    fclose($fp);

    i allready play with max_execution time in php and set_time_limit to 10 minutes.
    another think after that script start again it does about 100 thousand rows in database (i only have 80 thousand).
    Any help will be appreciated! thx [ed. note: electroid last edited this post 14 years, 7 months ago.]
      from Russia with love
      • 34012
      • 88 Posts
      I would recommend you to use direct CSV import through shell, but if that is not an possibility, you could use native PDO methods.

      Here's a test that I run earlier some time ago when topic was pretty much the same smiley

      https://forums.modx.com/thread/71759/populate-large-data-in-xpdo-solved#dis-post-402562
        Almost retired from web-development industry but still randomly writing at Lazylegs.info and on schedule hopefully in near future to finish Oracle and PostgreSQL ports of MODX
        • 38954
        • 24 Posts
        Thanks for your reply.
        I allready read your thread, but still beleave the way around this problem exist (so naive)). Another think i'm not very skilled in mySql, so i will try to do it with PDO.
          from Russia with love
          • 3749
          • 24,544 Posts
          I would try breaking the CSV into two or three files and do them one at a time.


          ---------------------------------------------------------------------------------------------------------------
          PLEASE, PLEASE specify the version of MODX you are using . . . PLEASE!
          MODX info for everyone: http://bobsguides.com/modx.html
            Did I help you? Buy me a beer
            Get my Book: MODX:The Official Guide
            MODX info for everyone: http://bobsguides.com/modx.html
            My MODX Extras
            Bob's Guides is now hosted at A2 MODX Hosting
            • 38954
            • 24 Posts
            Yes, BobRay, i think it will do the job.
            But i want be clear, so i rewrite my script to work with Pdo. But it doesn't work for me, maybe you can help me with some corrects on it. Code looks like this:

            $xpdo = new xPDO($dsn,'root','');
            echo $o=($xpdo->connect()) ? 'Connected' : 'Not Connected';
            $xpdo->addPackage('csvcars','C:/xampplite/htdocs/atlanticauto/core/model/');
            $manager = $xpdo->getManager();
            $manager->removeObjectContainer('csvCar') ? 'Removed' : 'Not removed';
            $manager->createObjectContainer('csvCar') ? 'Created' : 'Not created';
            $table = $xpdo->getTableName('csvCar');
            $sql = "INSERT INTO {$table} (lot, vin, image, saledata, year, make, model, place, odometer, bid, damage, repair, bodystyle, color, engine, drive, fuel, cylinders, acv, secdamage, keys, note, facility, saletime, salezone, salestatus, titlestate, titletype) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
            $q = $xpdo->prepare($sql);
            $fp = fopen("C:/xampplite/htdocs/atlanticauto/salesdata_250.csv", "r");
            $k = 0; 
            while (($line = fgets($fp, 4096)) !== false) {
                $fields = explode(',', $line);  
                $q->execute(array(
                    $fields[7],
                    $fields[21],
                    $fields[40],
                    $fields[2],
                    $fields[9],
                    $fields[10],
                    $fields[12],
                    $fields[35],
                    $fields[22],
                    $fields[33],
                    $fields[15],
                    $fields[25],
                    $fields[13],
                    $fields[14],
                    $fields[26],
                    $fields[27],
                    $fields[29],
                    $fields[30],
                    $fields[24],
                    $fields[16],
                    $fields[19],
                    $fields[34],
                    $fields[1],
                    $fields[4],
                    $fields[5],
                    $fields[32],
                    $fields[17],
                    $fields[18],
                ));
                echo $fields[7];
                echo "
            ";
            }
            fclose($fp);


            No errors, only empty database. I allready try with q->bindValue() and rewrite it to q->execute(array()), but still the same. Thx.
              from Russia with love
              • 3749
              • 24,544 Posts
              I would suggest going back to your original code, since that worked. wink


              ---------------------------------------------------------------------------------------------------------------
              PLEASE, PLEASE specify the version of MODX you are using . . . PLEASE!
              MODX info for everyone: http://bobsguides.com/modx.html
                Did I help you? Buy me a beer
                Get my Book: MODX:The Official Guide
                MODX info for everyone: http://bobsguides.com/modx.html
                My MODX Extras
                Bob's Guides is now hosted at A2 MODX Hosting
                • 38954
                • 24 Posts
                Maybe you're right and i'm just lasy to write split file function. )))
                  from Russia with love
                  • 38954
                  • 24 Posts
                  You know even after i make my file split in few, the problem still exist. It was easy to split, i just use class from here http://www.chang2chang.com/2010/06/split-text-file-by-line-numbers-in-php/. It looks for me like memory problem, my server just not enough strong to handle all data.
                  Should i make one snippet which split file and recreate table and another one which will create rows and save data in db and run second snippet from first (second snippet will only process database operations and first will hande all csv files)?
                  Thx.
                    from Russia with love
                    • 3749
                    • 24,544 Posts
                    I suspect that your problem is time, not memory or computing power. I think the PHP process is timing out (on many servers, you only have 30 seconds no matter what). You definitely shouldn't do everything in one bit of code.


                    ---------------------------------------------------------------------------------------------------------------
                    PLEASE, PLEASE specify the version of MODX you are using . . . PLEASE!
                    MODX info for everyone: http://bobsguides.com/modx.html
                      Did I help you? Buy me a beer
                      Get my Book: MODX:The Official Guide
                      MODX info for everyone: http://bobsguides.com/modx.html
                      My MODX Extras
                      Bob's Guides is now hosted at A2 MODX Hosting
                      • 38954
                      • 24 Posts
                      Ok, thanks for reply.
                        from Russia with love