We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
MODX Community Forums Archive

Can importX import 7,000 records?

    • 17249 ☆ A M B ☆
    • 165 Posts
    lithiumlab Reply #1, 13 years, 1 month ago
    I'm working in the migration of a products table where ill need almost all the fields in a Resource, so importX makes sense to me. Im trying to import nearly 7,000 records but only 488 gets imported. Nothing on the console. Any hints?
      • 33968
      • 863 Posts
      Lucas Reply #2, 13 years, 1 month ago
      Not sure but it wouldn't really take that long to break your import into chunks of 400 or so, right?
        • 18373 ☆ A M B ☆
        • 3,141 Posts
        Mark Hamstra Reply #3, 13 years, 1 month ago
        It imports whatever it can until php times out. It probably stops processing after 30 seconds, so try setting the time limit to like 30 minutes.

        Also, can you file a feature request on Github, to make ImportX attempt to set the limit on runtime? Thanks!
          Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

          Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.
          • 17249 ☆ A M B ☆
          • 165 Posts
          lithiumlab Reply #4, 13 years, 1 month ago
          @Mark;

          Thanks a lot for the info.
          I will submit the request.

          How can i modify the timeout?
          I'm really no php expert. smiley
          [ed. note: lithiumlab last edited this post 13 years, 1 month ago.]
            • 17249 ☆ A M B ☆
            • 165 Posts
            lithiumlab Reply #5, 13 years, 1 month ago
            @Lucas;

            Yeah probably wouldn't take long to brake apart the records but i prefer to know why can't.
            I prefer to dig in the issue a bit more to get consistence in the import process since i will use modx revo + importX for future projects.
            [ed. note: lithiumlab last edited this post 13 years, 1 month ago.]
              • 18373 ☆ A M B ☆
              • 3,141 Posts
              Mark Hamstra Reply #6, 13 years, 1 month ago
              You can add this in the snippet:

              set_time_limit(0)


              though there is no guarantee it will work. It wont work on IIS servers or when safe_mode is enabled, for one smiley

              The reason you can't do 7K at a time without that is cause the server typically limits how long a process can run, which is a good thing in case something spins out of control into an endless loop. I filed it as an improvement for a next version: https://github.com/Mark-H/importX/issues/25 (though I have no idea when I'll have time to work on that next version).
                Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

                Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.
                • 17249 ☆ A M B ☆
                • 165 Posts
                lithiumlab Reply #7, 13 years, 1 month ago
                I'm on IIS.
                I manually edited php.ini max_execution_time, restarted IIS and it worked.
                Thanks for all the info. Really helpful.

                Maybe its good idea to place a console error there in the meanwhile.

                ImportX rocks!
                Again, thanks a lot.

                Pablo.
                  • 18373 ☆ A M B ☆
                  • 3,141 Posts
                  Mark Hamstra Reply #8, 13 years, 1 month ago
                  If the process gets killed how would it return an error to the console..? laugh No idea how to do that.

                  Thanks for reporting how to change it on IIS.
                    Mark Hamstra • Developer spending his days working on Premium Extras and a MODX Site Dashboard with the ability to remotely upgrade MODX and extras to make the MODX world a little better.

                    Tweet me @mark_hamstra, check my infrequent blog at markhamstra.com, my slightly more frequent ramblings at MODX.today or see code at Github.
                    • 17249 ☆ A M B ☆
                    • 165 Posts
                    lithiumlab Reply #9, 13 years, 1 month ago
                    Just FYI, if you are running on FastCGI this are additional changes needed to be able to import large quantities of resources in Windows/IIS:

                    In my case 7,000+ resources took more than 10 mins but it worked flawlessly. smiley
                    Thanks a lot for such an awesome Extra.

                    Add this lines in fcgiext.ini on your system32/inetsrv.

                    [PHP]
ExePath=PATH_TO_YOUR\php-cgi.exe
ActivityTimeout=1800
RequestTimeout=1800


                    Also make sure max_execution_time in your loaded php.ini is the same.

                    *You'll need to change [PHP] to whatever your instance of php is named.
                    Hope this info can be useful for modxers using windows. Took me a lot of time to figure out.
                      • 17249 ☆ A M B ☆
                      • 165 Posts
                      lithiumlab Reply #10, 13 years, 1 month ago
                      It would be awesome to be able to export to a CVS based on a parent container or template ID.
                      The CSV can help a lot to carry on associated TVs in the same format it does importing and then be able to import in another site (matching new TVs values, of course or bulk updating TVs, aliases, etc).

                      Did a feature request at:
                      https://github.com/Mark-H/importX/issues/26

                      Related:
                      https://forums.modx.com/thread/71749/importing-resources-and-tv-content-to-2-1-3-pl-to-2-1-5-pl#dis-post-399889