Recommended workflow for maintaining sync across multiple site locations?

The scenario:

You have a site that you’ve been developing on your local workstation. The site is not finished, but it’s time to start getting outside feedback, so it needs to be pushed out to a remote server for others to access it.

The first time you do this it’s easy: Push the code up to the server, take a dump of your local MODx database, and load that onto the remote DB (then make any server specific config.inc.php changes that are needed).

The problem:

Once the site is on the remote server, you realize that certain adjustments need to be made to resources, snippets, etc, in order for the site to function. For example, you may have forgotten to publish some resources (and didn’t notice because you’re always logged in on your local box, and there’s sadly no indication of what’s been published or not on the UI ), or changes may need to be made to property sets of snippets - in my specific case, I needed to change formProcAllowedReferers in order for SPForm to function properly on the remote server. These are specific examples, but you can use your imagination to see that any number and type of changes made need to be made because of the change in server environment, depending on the complexity of your site.

Further, you know that in the future you’ll most likely want to push another complete code/DB dump to the remote server (since the site is still in early development) to re-sync your site versions.

The problem is that any changes you make on the remote server DB via the MODx manager will be overwritten the next time you push your DB. One way to avoid this is to first make the change on your local version then push another DB dump - OR - make the change in both places. Of course another way would be to keep a record of what was changed and where (having detailed knowledge of the DB schema), and manually push only the tables that needed to be updated on the remote site...or maybe make the changes remotely and push them back to your local before continuing development...

Obviously none of these solutions bodes well for streamlined ongoing development and maintenance of your site.

These sorts of things are easy to deal with when all of your code and configuration files are on the file system and/or under version control, but I’m wondering how have others dealt with this type of scenario with MODx? Is there an implied way to manage the dev-->staging-->live process for MODx? And what if there are multiple developers working on the same code base - how does that work?

One possible solution that comes to mind is adding the ability to push/pull selective resources/changes (property sets, chunks, snippets, etc) directly from a local MODx manager to a remote site. Further, some kind of heartbeat connection could optionally be maintained between MODx installations, so that users on any end of a connection will always know if/when they are in sync with the installation on the other side of the heartbeat, and with regard to which specific resources (some IDE’s with code publishing/FTP features do this). I’m not sure if this fits into the overall paradigm/vision, however...it also unfortunately means punching a hole in your remote firewall for outside Mysql connections (or needing ssh tunneling, vpn, etc)...

Or maybe it would be as simple as adding a way to selectively dump resource data to an intermediate file format (SQL, XML, etc) from within the MODx manager...

FYI, this is not an idle hypothetical question, I’m actually trying to work through this on an actual project - so any experiences or advice (particularly from the devs) will be greatly appreciated.

Please also let me know if I’ve missed something - is this what transport packages are for? If so, how do they fit into this particular day-to-day scenario?

Is this something that has already been considered for current MODx/Revo users, is it something still on the roadmap, or is this not at all a consideration for MODx development?

NOTE: underlines added to make my specific questions easier to read amidst the rest of the rambling.

One possible solution that comes to mind is adding the ability to push/pull selective resources/changes (property sets, chunks, snippets, etc) directly from a local MODx manager to a remote site

.

You can do this with Provisioner, but in your case you’d install it into your production server and point it at you’re ’remote’ site, which in your case is your staging server. You can now tweak individual resources/elements in production by ’importing’ them from the staging server. You can’t do property sets yet but that’s just cos its not implemented, not that its not doable. Its not automatic, its a GUI based tool so its not scriptable but does allow fine tuning so as to speak, really it assumes that you have aligned the packages between the two sites(it can help here as well) and you are just tweaking like you described above.

To be more global in answer I don’t think there is a ’standard’ way to do what you describe , not yet anyway, people seem to do there own thing here at the mo. Some people move entire sites etc.

BTW, there *is* an indication of publication status in the Resources tree, though I can never remember it and it’s kind of subtle.

Published - Roman type
Unpublished - Italic Type
Hide from Menus - gray
Show in Menus - black

Since I can’t seem to remember that (I think they should be reversed with Hide From Menus in sneaky italic type), I usually right-click and see whether the UI offers to Publish or Unpublish the resource.

You can do this with Provisioner, but in your case you’d install it into your production server and point it at you’re ’remote’ site, which in your case is your staging server. You can now tweak individual resources/elements in production by ’importing’ them from the staging server. You can’t do property sets yet but that’s just cos its not implemented, not that its not doable. Its not automatic, its a GUI based tool so its not scriptable but does allow fine tuning so as to speak, really it assumes that you have aligned the packages between the two sites(it can help here as well) and you are just tweaking like you described above.

Thanks for the info. Provisioner looks interesting, and I think it’s a step in the right direction. I’m wondering though, what the use case is for the tool - ie, under which scenarios would someone want/need to pull resources from a remote site?

I could probably use it in its current form to solve part of my issues, but since my "staging" server in your description is actually my laptop, there are some complexities which make the idea of accessing a remote server in order to round trip and access my laptop not quite what I had in mind. Then there are issues like dynamic IP’s (and the lack of a dynamic DNS setup) and security that I’m not sure I’m up for tackling to get Provisioner to work.

Obviously in my scenario I’d vote for making Provisioner a two-way tool. That way I can push changes from my laptop to a remote server for testing, and optionally pull down changes that were made on the remote server directly. Another use for this kind of functionality would be potentially to mimic the functionality of a code "repository". A master server (or virtualhost) could serve as the official version of a site, and then devs on a team can pull other devs’ changes down to their local boxes as needed. Then when the time came, the 2-way Provisioner could be used to push changes out from the repository to a production site (which would always be a one-way operation).

I’ve used a similar setup in conjunction with well-articulated policies on smaller collaborative projects where using an actual code repository was either impractical or impossible, with success.

Of course this would introduce problems like conflict resolution, etc...but I’m just thinking out loud...trying to understand if/how MODx can be used in a more complex/sophisticated development context which involves more than one site instance, etc.

BTW, there *is* an indication of publication status in the Resources tree, though I can never remember it and it’s kind of subtle.

Right you are! Wow, I actually stared at my resource list wondering if there was any visual difference between published/unpublished, and concluded that it was a missing feature. Now I see that it’s not missing, it’s just not really doing the job too well. Sneaky is right IMO the difference between published/unpublished should jump off of the screen - it should be easy to quickly scan a list of resources and understand publish state without thinking - either using icons or colors, and something more intuitive. Aside from the names/folders of resources, I think published state is the most important information you can get from a list of them in the space available.

Also, what does the icon mean? Will web resources always all have the same icon? (So far, I’ve only used vanilla web resources, and all the icons are the same) If so, the icon doesn’t serve a functional purpose, and should either be removed to make space for usable info...or it could be used to convey publish state. My 2c.

And maybe this is something that would make sense to be able to customize in system settings to suit your visual taste (but potentially overkill, and maybe that’s the job of a manager UI theme/skin).

Happy to put this in JIRA or someplace if devs think I wouldn’t be wasting time or bandwidth...but italics on those small fonts are just hard to see.

Quote from: stephenrs at Oct 22, 2009, 07:26 PM

Obviously in my scenario I’d vote for making Provisioner a two-way tool. That way I can push changes from my laptop to a remote server for testing, and optionally pull down changes that were made on the remote server directly. Another use for this kind of functionality would be potentially to mimic the functionality of a code "repository". A master server (or virtualhost) could serve as the official version of a site, and then devs on a team can pull other devs’ changes down to their local boxes as needed. Then when the time came, the 2-way Provisioner could be used to push changes out from the repository to a production site (which would always be a one-way operation).

You *really* should look into Transport Packages, which are built into the core of Revolution.

Quote from: stephenrs at Oct 22, 2009, 07:54 PM

IMO the difference between published/unpublished should jump off of the screen - it should be easy to quickly scan a list of resources and understand publish state without thinking - either using icons or colors, and something more intuitive. Aside from the names/folders of resources, I think published state is the most important information you can get from a list of them in the space available.

Deleted is probably more important. And the unpublished state has always been italics in MODx - we didn’t want to change it to confuse older users.

Also, what does the icon mean? Will web resources always all have the same icon?

No. Static Resources/weblinks/symlinks all have different icons. Resources with children have different icons, each Element type has different ones, and files have different icons depending on their filetype.

Quote from: splittingred at Oct 22, 2009, 09:47 PM

Deleted is probably more important. And the unpublished state has always been italics in MODx - we didn’t want to change it to confuse older users.

I’m an older user and I’m confused already.

But I agree they shouldn’t change. I wish I had been around when the initial decision was made.

Would it slow things down to introduce color (e.g. published in green or published + not hidden in green)? I agree that it would be nice to have something more salient for published resources than standard black roman type to make them stand out at a glance.

Thanks for the info. Provisioner looks interesting, and I think it’s a step in the right direction. I’m wondering though, what the use case is for the tool - ie, under which scenarios would someone want/need to pull resources from a remote site?

.

Yes, at the mo its just for tweaking last minute stuff when your packages are aligned between your two sites, not every single change is going to be packaged instantly, you may find a bug for instance and just want to transfer a single snippet quickly to get things working, you can do an official upgrade later say. I’m not sure at the mo how packaging would help in this scenario , we would have to extract our change set, package them up and install them on the production server but what namespace do we use? The tweaks could be all over the place for instance. A tool to make a packaged changeset would be nice I suppose.

It can be adapted to do most workflows I would have thought, like the one suggested above, we just don’t have any yet.

You can create batch-files for fast and easy syncing:

example:


or use something like http://www.heidisql.com/