Can importX import 7,000 records? - My Forums

Re: Can importX import 7,000 records?

lithiumlab — Nov 14, 2011, 03:24 PM

It would be awesome to be able to export to a CVS based on a parent container or template ID.
The CSV can help a lot to carry on associated TVs in the same format it does importing and then be able to import in another site (matching new TVs values, of course or bulk updating TVs, aliases, etc).

Did a feature request at:
https://github.com/Mark-H/importX/issues/26

Related:
http://forums.modx.com/thread/71749/importing-resources-and-tv-content-to-2-1-3-pl-to-2-1-5-pl#dis-post-399889

Re: Can importX import 7,000 records?

lithiumlab — Nov 14, 2011, 03:06 PM

Just FYI, if you are running on FastCGI this are additional changes needed to be able to import large quantities of resources in Windows/IIS:

In my case 7,000+ resources took more than 10 mins but it worked flawlessly.
Thanks a lot for such an awesome Extra.

Add this lines in fcgiext.ini on your system32/inetsrv.

[PHP]
ExePath=PATH_TO_YOUR\php-cgi.exe
ActivityTimeout=1800
RequestTimeout=1800

Also make sure max_execution_time in your loaded php.ini is the same.

*You'll need to change [PHP] to whatever your instance of php is named.
Hope this info can be useful for modxers using windows. Took me a lot of time to figure out.

Re: Can importX import 7,000 records?

markh — Oct 19, 2011, 10:52 AM

If the process gets killed how would it return an error to the console..? No idea how to do that.

Thanks for reporting how to change it on IIS.

Re: Can importX import 7,000 records?

lithiumlab — Oct 19, 2011, 10:37 AM

I'm on IIS.
I manually edited php.ini max_execution_time, restarted IIS and it worked.
Thanks for all the info. Really helpful.

Maybe its good idea to place a console error there in the meanwhile.

ImportX rocks!
Again, thanks a lot.

Pablo.

Re: Can importX import 7,000 records?

markh — Oct 19, 2011, 10:11 AM

You can add this in the snippet:

set_time_limit(0)

though there is no guarantee it will work. It wont work on IIS servers or when safe_mode is enabled, for one

The reason you can't do 7K at a time without that is cause the server typically limits how long a process can run, which is a good thing in case something spins out of control into an endless loop. I filed it as an improvement for a next version: https://github.com/Mark-H/importX/issues/25 (though I have no idea when I'll have time to work on that next version).

Re: Can importX import 7,000 records?

lithiumlab — Oct 19, 2011, 10:01 AM

@Lucas;

Yeah probably wouldn't take long to brake apart the records but i prefer to know why can't.
I prefer to dig in the issue a bit more to get consistence in the import process since i will use modx revo + importX for future projects.

Re: Can importX import 7,000 records?

lithiumlab — Oct 19, 2011, 09:48 AM

@Mark;

Thanks a lot for the info.
I will submit the request.

How can i modify the timeout?
I'm really no php expert.

Re: Can importX import 7,000 records?

markh — Oct 19, 2011, 03:58 AM

It imports whatever it can until php times out. It probably stops processing after 30 seconds, so try setting the time limit to like 30 minutes.

Also, can you file a feature request on Github, to make ImportX attempt to set the limit on runtime? Thanks!

Re: Can importX import 7,000 records?

okyanet — Oct 19, 2011, 03:20 AM

Not sure but it wouldn't really take that long to break your import into chunks of 400 or so, right?

Can importX import 7,000 records?

lithiumlab — Oct 18, 2011, 03:49 PM

I'm working in the migration of a products table where ill need almost all the fields in a Resource, so importX makes sense to me. Im trying to import nearly 7,000 records but only 488 gets imported. Nothing on the console. Any hints?