Updating from 0.9.x to 1.0

Here’s a proposal for the Updater, prompted by a query on the forums.

For 1.0 read "version with substantial db changes".

Basically I see the Updater as a separate project to 1.0 itself, comprising two parts:

A) 0.9.x Module: tests to find problem areas - for example, it may look to see if there are any modules other than the standard ones, and warn that these will need reinstalling and possibly rewriting. Basically sees if there is anything that the Updater Routine can’t handle.

B) Updater Routine: Goes through and migrates what data it can to the new format.

Other points:

1) These will come out *after* early betas of 1.0, ie we won’t be converting data until we’ve completely finalised what form that data needs to be in. Most other major PHP upgrades work this way.

2) It will be a distinct download, so that the 1.0 download is not bloated with code that is irrelevant to new users. The 1.0 installer should direct users to download the updater if required.

3) It will be a distinct project, with SVN etc.

4) It may have different project leaders to the 1.0 itself.

5) It will ideally fully convert content related to 0.9.7 snippets to equivalents in 1.0, with help from the snippet authors.

6) It may be possible for additional converters to be added in for other popular snippets. Perhaps say MaxiGallery.

Thoughts?

Sounds excellent!

Are there plans to start this project soon? Although the new structure isn’t yet known or finalized, some things are known already aren’t they? And the old structure certainly is. Perhaps some work could already be done for that part?

Well, this hasn’t really been approved yet.
But at this stage, I think we’d just be information gathering. Actually, some parts of this dovetail into the explanation of how things change, and if for example having chunks with the same names as snippets is going to be an issue, then we can encourage best practices now.

Actually, this will already be handled in a pretty straightforward manner with XPDO ... I’ll let Jason elaborate, but it shouldn’t be a big deal and a pretty seamless transition has always been an assumption during it’s entire development cycle.

The legacy-API API-extension makes sure that existing code works, using existing calls, too. Only time that it might break is if non-public API functions were used.

Interresting Ryan. I think I am going to dig into the xPDO website, as it seems much more important to the 1.0 architecture than I would have thought (simplifying : my take was that this enabled easy use of alternate databases like PostGreSQL, Oracle or whatever, and provided something similar to active records. But then again I am not skilled enough to understand ORM or CRUD in depth).

And Paul : great initiative and crystal clear/organized/insightful ideas as usual
I’ll be following this thread with interrest, though I am out of it skill-wise...

XPDO is the heart of the new core, actually ... not just an add-on nicety. And it’s a very technically advanced way to architect a system. It’s honestly probably best to hold out for the first test-build release and to start playing with it to see how it really ticks.

If we end-up doing a stepping-stone release (0.9.7), the 0.9.7 installer will basically be the same, but by installing it, you will also be installing the new xPDO Installation Manager that will help prepare the system for the leap to 1.0. Let me try to explain by showing the steps of new install for 1.0...

New Installation of 1.0
* User downloads bootstrap file and places it in a location on their webserver
* They launch the bootstrapper, which collects FTP user account information, does various configuration checks, and installs the xPDO core if it passes and you indicate you are ready (i.e. have backed-up everything properly)
* The bootstrapper then turns control over to the Installation Manager, which asks for the application you want to install; in this case, it would be a MODx 1.0 Core distribution, perhaps, let’s say, a blog distro.
* The installer asks for additional info, like db connection properties, and for the /core, /assets, and /manager directory names/locations
* The selected xPDO "Transport" packages are installed directly to the server locations specified; in this case it would be a core package, followed by several add-on packages that would include the blog functionality. The transport package contains everything it needs to install itself into any compatible deployment.
* Installed packages can have auto-update sites which can be saved into a deployment, and it will be checked from a manager UI where users can one-click update a specific package, be it the xPDO core, the MODx core, or any add-on packages.

If 0.9.7 installs the xPDO install manager itself, then the 0.9.7 to 1.0 upgrade can be handled completely via this same concept, only skipping the bootstrap step.

Wow. I, like David, thought xPDO was "just" the database API, not an entire platform.

So after having installed 0.9.7 I have:
MODx
xPDO Installation Manager (that wasn’t used to install MODx 0.9.7)
MODx database as per current structure.

Does this then mean that the 0.9.7>1.0 should be a particular MODx distribution which contains the same add-ons at 0.9.7 and is possibly the only time that it will additionally add legacy API methods using the new API?

(If we have multiple distributions, it would make sense to have one that doesn’t have legacy features).

So, I would somehow provide xPDO with the upgrade details (do I download a zip and put it somewhere, or does xPDO download the app for me?)
and it would ditch the non-xPDO-installed-but-using-xPDO-api MODx 0.9.7,
convert the structure as necessary,
install MODx 1.0
and make everything work?

Is help needed with this bit, or is it all sorted?

I still think there’s a benefit in having a "Readiness Checker" module - possibly it would recommend which distribution you’d need to install based on what you’re actually using. I am certain that people will feel more confident upgrading if they could verify in advance whether there were going to be issues.

Now, I understand that "bootstrap" is a logical and correct name for a pre-installer, but it’s still jargon. Basically you install xPDO and then you install MODx from within xPDO. If you already have xPDO you don’t need to reinstall it. I’d therefore suggest that the bootstrap is just called the xPDO Installer, and to avoid confusion the xPDO Installation Manager could become the xPDO Application Manager.

Quote from: PaulGregory at Nov 08, 2006, 01:26 PM

Wow. I, like David, thought xPDO was "just" the database API, not an entire platform.

So after having installed 0.9.7 I have:
MODx
xPDO Installation Manager (that wasn’t used to install MODx 0.9.7)
MODx database as per current structure.

Does this then mean that the 0.9.7>1.0 should be a particular MODx distribution which contains the same add-ons at 0.9.7 and is possibly the only time that it will additionally add legacy API methods using the new API?

(If we have multiple distributions, it would make sense to have one that doesn’t have legacy features).

Well, there would be an installation site let’s call it, where all the packages are stored (could be a location on an FTP, web server, or even Amazon S3). My plan would be to have a manager interface within MODx that uses the xPDO application provisioning classes and the bootstrap installer infrastructure to check all your currently installed packages against the latest version of those packages on the installation site. If a newer version of a package (either xPDO itself, the MODx core, or any add-on package installed) is available, it would then allow a one-click direct to server upgrade or provide instructions on how to download the package and put it in the proper location to invoke the xPDO installer process on it.

So for the 0.9.7 path, I would offer a specific upgrade distribution meant to support as much legacy code as possible, though the 0.9.7 to 1.0 path will likely be much less tolerant of legacy code, especially legacy code that directly queries the database or depends on the current database structures. New installations could choose the package without legacy support included, which could also be offered as an add-on package.

Quote from: PaulGregory at Nov 08, 2006, 01:26 PM

So, I would somehow provide xPDO with the upgrade details (do I download a zip and put it somewhere, or does xPDO download the app for me?)

The bootstrap process Victor has been working on can handle direct server-to-server installation, though we will of course offer manual alternative steps where you can download the packages and manually place them on the server. The xPDO bootstrap step would require the download and placement of a single file and would initially ask for your FTP/user account details and the location to install the xPDO core (which BTW, does not have to be in the web root at all, so it could be used by many accounts theoretically).

Once xPDO core is in place, the installation would then ask for an application package, in this case, we would offer the various MODx distributions, and once selected and your database credentials are provided the package would be installed by the xPDO installation manager, including any additional interactive steps which the package itself can define (i.e. asking for the name/location for your assets and manager directories).

Quote from: PaulGregory at Nov 08, 2006, 01:26 PM

and it would ditch the non-xPDO-installed-but-using-xPDO-api MODx 0.9.7,
convert the structure as necessary,
install MODx 1.0
and make everything work?

That’s the idea...we’re even discussing offering an optional direct to Amazon S3 backup solution so automatic backup and restore options can be enabled during the one-click install process. Might be a good cheap extra service to provide to help fund the project.

Quote from: PaulGregory at Nov 08, 2006, 01:26 PM

Is help needed with this bit, or is it all sorted?

I still think there’s a benefit in having a "Readiness Checker" module - possibly it would recommend which distribution you’d need to install based on what you’re actually using. I am certain that people will feel more confident upgrading if they could verify in advance whether there were going to be issues.

Now, I understand that "bootstrap" is a logical and correct name for a pre-installer, but it’s still jargon. Basically you install xPDO and then you install MODx from within xPDO. If you already have xPDO you don’t need to reinstall it. I’d therefore suggest that the bootstrap is just called the xPDO Installer, and to avoid confusion the xPDO Installation Manager could become the xPDO Application Manager.

I need all the help I can get making this come to life, but Victor and I have gotten the ball rolling. We’ve already got the proof of concept bootstrap installer, though it needs a bit of refactoring, as it was designed against legacy MODx assumptions for directory structures, etc. I’m coding the foundation of the xPDO application package installer piece, which boil down to 2 base classes, an xPDOTransport object representing the physical package and it’s metadata, and xPDOVehicle objects. These vehicles wrap a single installation entity, be it a record from a database object, a physical file, or a PHP or SQL update script that represents upgrade requirements between a single revision of the package (i.e. upgradeJot_from0.9_to1.0.php OR upgradeJot_from1.0_to1.1.php).

xPDO Installation and Application Provisioning, hmmm, I like it...though I’m hoping, once installed initially, the xPDO Installer is gone and even xPDO itself can be upgraded from the Application Manager via a "Transport" package.

BTW, some of this is based on how the Eclipse Installation and Configuration Management pieces work, where you can supply update sites for the core and various add-ons, and it can keep all the packages up to date, or roll you back to any previous configuration you have installed.

Wow. I, like David, thought xPDO was "just" the database API, not an entire platform.

It’s definitely not just a database API; it’s also a very handy way of loading classes in PHP 4 and 5, and as such, just happens to make a good core library manager for database-centric applications, and the only real requirement to kickstart the work of installing the individual pieces.

And actually, let me bullet point this bootstrap install process out to be clearer after re-reading and giving it a little more thought; and let’s focus on xPDO being a core part of MODx, and leave xPDO out of the related installer terminology, i.e. MODx Installer and MODx Configuration Manager rather than xPDO Installer and Application Manager.

[*] User downloads the bootstrap installer file, puts it in their web server where they will be installing MODx, and points their browser to it.
[*] User triggers the MODx Installer process.
[*] Installer begins by asking for acceptance of the license, etc. and then prompting the user for their FTP user account credentials to be used for the installation.
[*] Installer verifies the credentials and proceeds if valid.
[*] Installer connects to the main MODx download repository (we’re seriously thinking Amazon S3) and presents the user with a list of the latest core MODx package releases available. They select a package and installation continues.
[*] Installer prompts user to select a directory where the MODx core will be installed (should present a directory explorer of the FTP server). NOTE: This directory does not have to be within the webserver, just accessible to the PHP and FTP user accounts.
[*] Installer performs a series of checks on the configuration to determine a mode of installation (i.e. transfer a zip and extract it, or transfer all files individually via FTP).
[*] If checks pass for one of the valid modes, just the core files needed to start the package installation process are installed directly to the server; otherwise, alternate manual steps are described which the user can proceed with, but this should all work automatically in all but extremely rare configurations.

At this point meta data about the selected package is used to define these next steps, so in essense, the bootstrap process now passes control to the installed xPDO core resources to perform the rest of the actual MODx core package installation...so for example the first two steps in the MODx core package installation might be

[*] Installer prompts user to select a directory where the MODx web context will be installed (this is the location where the main index.php or front controller will be located), along with a name for the assets directory if they don’t want it to be called assets (optional). NOTE: These directories must be within the web server document root.
[*] Installer prompts user to select a directory where the MODx manager context will be installed (this is the location where the manager index.php or front controller will be located). NOTE: This directory must be within the web server document root, and in a different location than the web context.

So on and so forth... the installer just follows the manifest of the package, which prescribes an order to execute the transport vehicles I described. Each individual vehicle in the manifest can then do just about anything, prompt the user for input, create a data table, add a data object stored in a db portable serialized format (JSON in my early implementation), execute a SQL statement or two to migrate data to a new data structure, represent a file by moving it from a local package or directly from the update server, to the appropriate location in your site, etc.