Recommended workflow for maintaining sync across multiple site locations?

Yes buts that’s only the database, it doesn’t sync files between the two sites such as css files etc. It also doesn’t help people who use a flat file structure where all their snippets and stuff are ’hooked in’ at run time so the snippet in the db just contains this ’include <path to real snippet><name.php>’ often used so people can version stuff in the vcs of their choice.

Its quite a complex area this, I doubt if we will ever write a tool that can do everything for everybody no matter what workflow they may have in mind. We are probably better using the *nix approach whereby loss of smaller tools can be used in any combination to achieve the desired effect. As an example :-

How do you count the number of lines in a text file on linux from the CLI.

1. Use a specific tool that does this for you
2. Do something like this ’cat README.txt | wc -l’ ie use a combination of already existing tools together.

It might be possible to write a tool that would generate a transport package script (which could easily transport the files as well).

If db tables and files at the site had a "dirty" flag of some kind, it could search the whole site for things that were "dirty" and present them to you as a list with checkboxes, then clear the dirty flag for the ones that get tranferred.

In theory, you could move changes both ways with it, but to be a complete solution it would probably have to do date/time comparisons for everything and, with server time offsets and daylight savings time variations per state, it could get pretty hairy.

As you say, it’s definitely not a trivial problem.

Quote from: splittingred

You *really* should look into Transport Packages, which are built into the core of Revolution.

You’ve mentioned this a number of times before in other threads, but I still have little idea what you mean regarding transport packages...and I have so far not had the time to go through the transport package docs (assuming they exist). I guess I still need a some convincing, because so far my understanding of transport packages is that they are a convenient way to package up 3rd party components for distribution, but would perhaps be a cumbersome way to manage a project on a day to day basis - given other alternatives.

If you could briefly describe (perhaps list a few workflow steps) how you would envision transport packages playing a role in the specific scenario described on this thread, it sure would go a long way toward my understanding what they are for in a larger sense...and it would probably also serve to disseminate this info to wider audience, which would appear to be good for all, across the board. RTFM just doesn’t seem like the best way to go on this one.

Update: I decided to do my part by reading the manual section on transport packages, including the tutorial on creating a 3rd party component build script, but I’m not sure I fully get it.

What I can glean from the docs conceptually is that transport packages can provide a mechanism for getting MODx "stuff" from one place to another, but how to do this in real world practice, particularly as it applies to day to day maintenance isn’t clear.

So far, in terms of a maintenance workflow in the context of this thread, I can envision:

[ Assuming you already have a matching default MODx instance on your remote server, and have optionally setup your own custom ’provider’ ]

1. Create a transport package and build script for everything in your local dev environment (property sets??, files, chunks, snippets, system settings, etc).
2. Install the package on your remote system (or use your custom provider)

[ Then, something changes on the local ]

3. Update the build script to reflect the changes
4. Reinstall the package on the remote

[ Then, maybe, something changes on the remote that needs to get back into dev ]

5. Update the build script to reflect the change
6. Install the package locally

I can see how this might work - but only if some level of automation was injected, such as the ability to have the package management section of the MODx manager provide a layer of abstraction to build script management, and maybe the ability to push, in addition to pulling transport packages - perhaps to a provider (or maybe SVN would be better). Otherwise, manually maintaining a build script becomes a bottleneck, particularly for minor adjustments. (EDIT: Just realized this is what BobRay was talking about in his last post...now his post makes more sense to me...and yes, this all does open up a can of worms).

Is this the workflow you’re envisioning? Or am I missing something?

Can property sets be represented in build scripts? Is there anything in a MODx db that can’t be in a transport package?

Also, for the parts of a build script that pertain to replicating the state of data from one MODx install to another, can you explain why you didn’t decide to use something simpler, like XML, for example?

Quote from: splittingred

Deleted is probably more important.

Hmm...why? This doesn’t compute at all for me, please explain the rationale.

Quote from: stephenrs at Oct 23, 2009, 10:41 PM

I can see how this might work - but only if some level of automation was injected, such as the ability to have the package management section of the MODx manager provide a layer of abstraction to build script management, and maybe the ability to push, in addition to pulling transport packages - perhaps to a provider (or maybe SVN would be better). Otherwise, manually maintaining a build script becomes a bottleneck, particularly for minor adjustments. (EDIT: Just realized this is what BobRay was talking about in his last post...now his post makes more sense to me...and yes, this all does open up a can of worms).

Is this the workflow you’re envisioning? Or am I missing something?

Can property sets be represented in build scripts? Is there anything in a MODx db that can’t be in a transport package?

Also, for the parts of a build script that pertain to replicating the state of data from one MODx install to another, can you explain why you didn’t decide to use something simpler, like XML, for example?

This is basically the workflow we are envisioning (with the exception of minor adjustments in the script would be minimal since the script would be written to grab the latest information and would only need to be updated if the scope of the data/files involved in a package changed) and automation is absolutely part of the plan and why we are developing this as a scriptable API, but automation is much like optimization in my view, you do it last.

And yes, any data object in the MODx core, or even in your custom extensions can be represented in transport packages and manipulated through scripting, be it build scripts or whatever kind of scripts we’re talking about (i.e. xPDOVehicle validation and resolver scripts).

We already tried to build a manager UI to create the transport packages, but it got pretty complex so we tabled it until we at least get to an RC status, and until we got more feedback and experience with using our workflow in real projects using the xPDO/modTransport API’s via these build scripts. BTW, this interface produced and consumed XML configuration files describing the build process, so XML, YAML, or whatever could be used to drive this automation.

In addition, we are working on creating additional xPDOVehicle class types, and you can create your own. For instance, so transport packages can be embedded in other transport packages, an xPDOTransportVehicle class will handle the embedded xPDOTransport instance including calling it’s installation process; this will be great for combining pre-existing and new custom packages. Another involves generating SQL from the contents of a Vehicle for installing/upgrading large quantities of data objects; this will allow for much more efficient installations/upgrades than can be achieved with the xPDOObjectVehicles directly, though might not be practical in some situations where complex data relationships need to be resolved. I can even envision an xPDOSVNVehicle that used an SVN API to put content in/get content out of an SVN repository in various ways.

Just my 2 cents here but this is potentially a very large undertaking no matter how smart the transportation layer is.

We already tried to build a manager UI to create the transport packages, but it got pretty complex ...

.
I can imagine, I’ve seen automated package builders before for packaging into RPM format, they are OK for fairly simple stuff but you couldn’t use one for packaging something like PHP for instance, if you understand what you need to do in these cases its quicker to write the spec file by hand, in our case the build.transport script of course.

The workflow is crucial here, there’s a difference for instance in stuff you package and stuff that’s supplied to you, if you own a package it’s relatively easy, if you don’t then what happens next? Say you’ve found a minor css bug in a 3PC or core extension, its a quick easy fix. Ok, you fix it, you need to do this to get your production site working, no choice here. This should now be pushed upstream so the owner can look at it, decide whether to adopt it or not and re-issue the package or not. You could re-package it yourself but its now out of step with upstream(could even raise a JIRA here as part of the workflow if needed, doesn’t help for 3PC’s with no JIRA though). If upstream pushes a new package without your change you can’t adopt it without re-applying your fix(patch).

This all needs to be tracked intelligently or it will be lost in the noise. These are layers over and above how transport packages work, we are into package management in general now. If we allow packaging of disparate items, i.e stuff that’s either not packaged that’s changed or tweaks/bugfixes as described above into ’changesets’(patches?). Ok, we can apply the changeset to production, fine , but what about the next one?. We would need to track not only that the changeset has been applied(we do this now) but what was in it in case the next one changes it(deliberately or otherwise). We are now into dependency resolution, aargh!!

For instance, so transport packages can be embedded in other transport packages..

Definitely needed, we should be able to do this fairly quickly even now, and install ’bundles’ of packages together, eg demo content.

Quote from: shamblett at Oct 25, 2009, 03:47 AM

We already tried to build a manager UI to create the transport packages, but it got pretty complex ...

.
I can imagine, I’ve seen automated package builders before for packaging into RPM format, they are OK for fairly simple stuff but you couldn’t use one for packaging something like PHP for instance, if you understand what you need to do in these cases its quicker to write the spec file by hand, in our case the build.transport script of course.

This is why we want to offer the API as well as provide builders for simple/known tasks. And well, most Extras are not anywhere near the scope of something like PHP. There will be patterns emerging for common tasks that can certainly be automated and turned into configuration file consumers/producers with simple user interfaces.

Quote from: shamblett at Oct 25, 2009, 03:47 AM

The workflow is crucial here, there’s a difference for instance in stuff you package and stuff that’s supplied to you, if you own a package it’s relatively easy, if you don’t then what happens next? Say you’ve found a minor css bug in a 3PC or core extension, its a quick easy fix. Ok, you fix it, you need to do this to get your production site working, no choice here. This should now be pushed upstream so the owner can look at it, decide whether to adopt it or not and re-issue the package or not. You could re-package it yourself but its now out of step with upstream(could even raise a JIRA here as part of the workflow if needed, doesn’t help for 3PC’s with no JIRA though). If upstream pushes a new package without your change you can’t adopt it without re-applying your fix(patch).

First, any Add-On or Core Extension hosted in our Extras can use JIRA, this is why we have the generic projects for each. And how a developer chooses to manage their project and distribute their code is up to them. I don’t see how this has anything to do with packaging; these same issues exist without packaging, only we leave it to the MODx user to resolve manually. At least with packaging we can provide easy ways for users to install/upgrade/uninstall things with some intelligence and without them having to read a small book of manual instructions.

Quote from: shamblett at Oct 25, 2009, 03:47 AM

This all needs to be tracked intelligently or it will be lost in the noise. These are layers over and above how transport packages work, we are into package management in general now. If we allow packaging of disparate items, i.e stuff that’s either not packaged that’s changed or tweaks/bugfixes as described above into ’changesets’(patches?). Ok, we can apply the changeset to production, fine , but what about the next one?. We would need to track not only that the changeset has been applied(we do this now) but what was in it in case the next one changes it(deliberately or otherwise). We are now into dependency resolution, aargh!!

I mean if you use MAMP and a PHP release comes out, you don’t just manually add the new version of PHP to MAMP, you wait for MAMP to release a new package that is known to work as a whole distribution. If combining packages you obviously tested the combination to work before distributing it. If you need to update something in the aggregation, well, you regression test all of the pieces and how they work together. I still don’t see how this has anything to do with whether the component is distributed in a package or manually cut/pasted into a distribution; the need for dependency resolution between disparate components already exists. And I’d sure rather be able to handle dependency resolution in a script I can attach to a new version of a package than have to write complicated manual instructions on how an end-user can resolve these himself.

If we follow the simple, reusable component philosophy when developing our packages, the patterns will emerge for automation and improvement, including how to handle dependency resolution.

At least with packaging we can provide easy ways for users to install/upgrade/uninstall things with some intelligence and without them having to read a small book of manual instructions.

Yes, absolutely.
I’m not saying don’t use packaging, we should definitely use packaging, its a big plus and a big step in the right direction. I’m just saying that the packaging API/scripts/transport classes themselves can’t implement a package management scheme alone, they need augmenting with a ’packaging layer’ that handles packages themselves, not how they are created and distributed.

Trivial example of where I’m coming from a bit.

There is a bug in a core file that stops your site working as you want it, OK, not good and maybe not that usual but it happens. You JIRA it, fair enough, you identify and propose a fix on the JIRA, trial it and it works. What do you do now to fix your site, which must be fixed right now?

Hand modify the affected file on your production server, this is probably what most if not all people do now.

How would packaging help here? Are you saying package up the whole core from your dev box and apply it on production?

I’m probably not understanding exactly how you intend to use packaging as suggested earlier in the thread to maintain a kind of push/pull automatic relationship between a dev box and a production server(s) or going from the thread title ’Recommended workflow for maintaining sync across multiple site locations?’ how it helps here.

Its not packaging itself, its exactly how we intend to use it, manage it, above how we use it now I’m getting confused about.

And I’d sure rather be able to handle dependency resolution in a script I can attach to a new version of a package...

I think this would be impossible to do, how would you write the script?. You don’t know the install environment till its installed. Then of course what if we can’t resolve? The ever perennial question, rollback, fail or what.
This again is management, not packaging as such.

If we follow the simple, reusable component philosophy when developing our packages, the patterns will emerge for automation and improvement, including how to handle dependency resolution

Yes agreed, but that’s just for us(’our’, above), 3PC’s can do what they like, package or not as they like and people can use these if and when they like. This is the real world, we can’t just assume we can automate installation and resolve dependencies just by writing smart install scripts, it won’t work. I’d rather not get into this area, if a user installs/upgrades a package that trashes several local edits he had in the package then so be it, we could help in this case by ’diffing’ the package to be installed with what’s installed, but this again is management, not packaging itself.

There’s been few threads lately where it’s been suggested that ’packaging’ can help/solve/make easier some process or other. I’m not sure it can in these cases or ever will be able to without significant effort.

What about this?

Rewrite SVN (or tap into it) so it will handle MySQL tables as well as files (merge would be a challenge ). Then you could have a repository somewhere and both the local install and the production server would be working copies.

Any changes could be committed to the repository and then moved to the other site via SVN upgrade (or merge, if necessary). TSVN shows you the changed items with checkboxes during the commit so you could choose to commit all or just some of the changes. Certain files and tables (e.g. manager logs, .htaccess) could be unversioned or set to SVN ignore.

Even without the MySQL component, it still might be feasible if you created an independent tool to to export or import the tables in the DB as separate .SQL files

Talk about a killer app.

I wish I had time to look into it further.