Migrating to Revolution

Quote from: BobRay at Jan 24, 2008, 10:19 PM

Do you think a simple GUI front-end for this would be worth the effort?

Excellent idea!

Also, you might be interested in how to do this in the 097 manager with Ext2. We’re putting up a documentation/tutorial site for this:

http://docs.modxcms.com/modext/

It’s in-progress, but should help a lot with how to use modExt (MODx’s Ext implementation) properly and effectively.


Also, you’ll probably want to look into modAction and modMenu for creating the controllers and menu items for the section.

Are we thinking an inside the manager interface is the right way to present this? i.e. install the new MODx and then run the migration script on an old database/file-system location?

Jason I think that sounds like a great idea in getting the site up and "proving" the core functions. Then the old site could be "imported". Would be pretty darned cool and also facilitate sucking in those other types of sites as you mentioned here/elsewhere ... slurp up your Wordpress blog entries under a specific container anyone? Ready to migrate from Joomla/Nuke/whatever? Fill out your DB credentials in this form, or locate your SQL dump and press this button ...

Quote from: OpenGeek at Jan 25, 2008, 09:18 AM

Are we thinking an inside the manager interface is the right way to present this? i.e. install the new MODx and then run the migration script on an old database/file-system location?

I was thinking standalone app., since the user will have to go outside of MODx to dump the SQL from the old site. It could certainly be done in the Manager but that would slow development some and make debugging more difficult.

If it works as a standalone, could it be integrated into the Manager later?

Bob

Quote from: BobRay at Jan 25, 2008, 09:50 AM

I was thinking standalone app., since the user will have to go outside of MODx to dump the SQL from the old site. It could certainly be done in the Manager but that would slow development some and make debugging more difficult.

There is no SQL dump involved. Everything in this script is done via xPDO objects or directly to files. And it could be distributed separately, but would be useless without the core install available and running and the app configured to point at the core location.

Quote from: OpenGeek at Jan 25, 2008, 10:02 AM

Quote from: BobRay at Jan 25, 2008, 09:50 AM

I was thinking standalone app., since the user will have to go outside of MODx to dump the SQL from the old site. It could certainly be done in the Manager but that would slow development some and make debugging more difficult.

There is no SQL dump involved. Everything in this script is done via xPDO objects or directly to files. And it could be distributed separately, but would be useless without the core install available and running and the app configured to point at the core location.

I was referring to the dump of the old site to be migrated. The script isn’t intended to go and extract the old site content is it? And, as a user, I’d be more comfortable having the program work on the SQL dump files and then taking a look at them before importing them into the 0.9.7 site.

I didn’t see anything in the code that looked like it required the core install to be *running* for it to work, just available. It looked to me like the script created its own instance of $modx in convert.php:



Anyway, I’ll start looking at developing it inside the Manager if that’s the prevailing thinking. I was hoping I could put off learning about the ext stuff a little longer

Bob

Ok, let’s step back. There are a large number of ways for this to be dealt with, but here is the vision I had...

[*] You install and configure 0.9.7
[*] You grab the 0.9.6 migration pack from a manual download site, or via the Workspace manager (remote download/install)
[*] The migration pack includes the relevant interfaces for the manager or a standalone web interface (maybe a context?), along with an xPDO model representing the 0.9.6 tables (or a process to generate the model for them using the xPDOGenerator API’s)
[*] You access the interface, select the classes that represent the tables you want to convert (probably with a recommended set of defaults), along with any directories/files you want to convert.
[*] It provides an interface to specify and review dry-run’s recorded to the log
[*] It provides the actual command to perform the translation and does it.

The question becomes, do we use the translator as is and have it update the old tables, then perform an import, or do we combine these two steps and have the translator apply to the table content as we move it to 0.9.7 structures.

Quote from: OpenGeek at Jan 25, 2008, 12:05 PM

Ok, let’s step back. There are a large number of ways for this to be dealt with, but here is the vision I had...

Good idea (stepping back, I mean).

[1] You install and configure 0.9.7
[2] You grab the 0.9.6 migration pack from a manual download site, or via the Workspace manager (remote download/install)
[3] The migration pack includes the relevant interfaces for the manager or a standalone web interface (maybe a context?), along with an xPDO model representing the 0.9.6 tables (or a process to generate the model for them using the xPDOGenerator API’s)
[4] You access the interface, select the classes that represent the tables you want to convert (probably with a recommended set of defaults), along with any directories/files you want to convert.
[5] It provides an interface to specify and review dry-run’s recorded to the log
[6] It provides the actual command to perform the translation and does it.

[7]do we use the translator as is and have it update the old tables, then perform an import, or do we combine these two steps and have the translator apply to the table content as we move it to 0.9.7 structures?

1 - no question about this
2 - or the various migration packs are install options (maybe later)?
3 - ok
4 - I’ve got some confusion here, but some ideas about it below.
5 - ok
6 - ok
7 - I would think the former for starters. We could always add the option to automate the import later.

Here’s what I’m thinking, based on your vision.

The user:

1. Gets the migration pack (manually as a first step, then, in a future version, in the Workspace Manager, and/or during installation)..
2. Gets both directory structures (0.9.6 and 0.9.7) on the same machine in different subdirectories.
3. Does an SQL dump/export of the old site and puts it in the root of the 0.9.6 MODx directory referred to in step 2.
4. Runs the migration script and selects the tables in the DB to be translated or moved (with sensible defaults and some things off limits).
5. Clicks to execute the script that copies all the selected tables in the big SQL dump file to a new .sql file, translating as it goes.
6. Gets the option to view (and edit?) the new .sql file (that’s the dry run for this step).
7. Imports the new .sql file into the 0.9.7 DB. (or we do it for the user?).
8. Specifies the two root directories and then specifies the parts of the 0.9.6 directory structure to be copied/translated/etc.
9. Gets the option to see a window with a dry run of the directory/file conversion/copy.
10. Ok’s the execution of the directory/file conversion/copy.
11. Gets down on his or her knees and thanks us for the opportunity to use the migration pack.

I’m seeing steps 4-7 on one screen and then 8-10 on another to keep the user from being overwhelmed by the complexity.

Step 8 is the monster, IMO, especially since we don’t know what directories the user has, and it will be hard to know ahead of time which directories should be merged, overwritten, deleted, or created. Also, which directories contain files that need translation (can we just offer to translate any files that end in .php?).

Maybe a big directory tree with check boxes next to each item (copy, merge, ignore) and a separate "translate .php files" box? Do we also need a "copy to location" choice?
Maybe two directory trees, one with stuff we know for sure what to do with and one with everything else (or the ones we know what to do with are grayed out in the one big tree?)

Or is this *way* too ambitious? Rahul says MODx can do air traffic control for the space shuttle. Is this the moral equivalent?

Bob

Anyway, I’ll start looking at developing it inside the Manager if that’s the prevailing thinking. I was hoping I could put off learning about the ext stuff a little longer

Bob

Hope some work Shaun’s been doing helps there: http://docs.modxcms.com/modext/

The ideas being discussed above sound really great. For large SQL files that might time out on a busy server, we might think about breaking it into chunks and reloading the page between chunks. SMF handles its DB maintenance operations this way and I’ve seen some operations take upwards of 5 minutes to complete but never have seen them timeout (due to the reloads and spawning new operations for lack of a better way to phrase it).

Alright, 50 lashes to the next person that mentions SQL files. Forget SQL files, they will not be involved here at all. And yes, this would need to be broken up into smaller, practical pieces if run through a web UI.