how many people are using modx on separate development & production servers?

hi,

i’m just wondering how many people are using modx in an environment where they develop on a development server, perhaps stage it on a staging server and then upload it to a production box.

i’m also curious how people are dealing with managing the testing & upload process, whether you use svn, whether you save database tables in a repository, and what tools you’ve created or use to help manage this.

do people think this is a gap in the current modx offering, the lack of tools to use the platform in a more corporate/managed environment?

redman:

We use MODx like this in several different scenarios, and we use Subversion for development and deployment in conjunction with hand-crafted SQL update scripts to manage the promotion workflow. We create one branch for each server in the workflow and merge as needed, then prepare for deployments by coordinating the merges with associated SQL scripts.

It is a gap in the offering for more formal environments like this, and one that Revolution is being developed to address via Transport Packages; these will offer convenient ways to move data and files between deployments.

The site I work on consists of documents created by users, admins and developers. We also have some more sites in the pipeline, so we have mutliple sites to manage.

Some of the problems we’ve faced are:

1) synchronising document ids on production and development boxes - if a developer is working on a new feature, and they need to create documents and link to them, then they need to first create a document on the production box, and then create a document with the same id on their dev box.
2) saving snippets, chunks and templates in svn,
3) coming up with a robust way to stage the sites.
4) regression testing

Before working on a new feature, devs copy the trunk to a branch in their directory. They check this out, and then run a phing build script that performs various actions such as purging their dev db, importing the latest nightly db snapshot, and importing sql files stored under svn.

To solve the first problem, I’m working on a client/server application. The server runs on the staging machine, which devs only have limited access to. Now, when they want to create a new document, they’ll run the client which will contact the server. The server will contact the production db, create a new document, return the new id to the client, and the client will then create a document with the same id in the developer’s database locally.

This has taken me a while to make.

Would the new transport package system provide a solution for these issues?

I’m wondering whether it’d be useful to create some documentation, best practices and tools like I’ve mentioned above to speed up development of sites in more formal environments, or whether people think each site/project is too unique to do this.

Quote from: redman at Sep 01, 2008, 10:53 AM

The site I work on consists of documents created by users, admins and developers. We also have some more sites in the pipeline, so we have mutliple sites to manage.

Some of the problems we’ve faced are:

1) synchronising document ids on production and development boxes - if a developer is working on a new feature, and they need to create documents and link to them, then they need to first create a document on the production box, and then create a document with the same id on their dev box.
2) saving snippets, chunks and templates in svn,
3) coming up with a robust way to stage the sites.
4) regression testing

Before working on a new feature, devs copy the trunk to a branch in their directory. They check this out, and then run a phing build script that performs various actions such as purging their dev db, importing the latest nightly db snapshot, and importing sql files stored under svn.

To solve the first problem, I’m working on a client/server application. The server runs on the staging machine, which devs only have limited access to. Now, when they want to create a new document, they’ll run the client which will contact the server. The server will contact the production db, create a new document, return the new id to the client, and the client will then create a document with the same id in the developer’s database locally.

This has taken me a while to make.

Would the new transport package system provide a solution for these issues?

In some ways yes though the first problem is tricky. We’ve been forcing all content changes through a development site, preparing the promotion through SVN and SQL and then promoting to QA then repeating for production. The content being managed on production is complicated, but I like your ideas for dealing with it. Tools to reduce or eliminate any need for SQL interaction is what I’m going for with the transport packaging concept, but I’m stuck on the problem of document id’s, which are coded into the content in the form of link tags. Perhaps we can combine the idea of the client/server app and the transport packages into an intelligent tool for these types of environments, though I’d specifically like to see it possible without the Phing requirement.

With the advent of transport packages, I have a way to create an installable aggregation of database changes that can be combined with file changes, but this does not specifically deal with Subversion. I’m working on a separate way of dealing with deployment of MODx itself via Subversion, but it does not address the issue of chunk and template content being stored in Subversion. You could keep the transport packages in Subversion, but handling all the database managed content as files is a separate effort in the Revolution core that still has another phase to go after the initial 2.0.0 release. That said, I currently keep snippets as includes on the filesystem, and use a single snippet to include file versions of any snippets and pass through the snippet parameters. This way, all my snippets are managed on the filesystem and that part of the database never changes. But this is less than ideal. Being able to store any Element’s content in Revolution (and migrating the content from site_content to an Element itself and relating it to a Resource/Document) on the filesystem is the solution I’m striving for. At that point, applying Subversion can be done in numerous ways, and we will also be able to use native versioning capabilities as an alternative core extension (i.e. have cake, eat it too).

Staging and regression testing are more complex subjects, but I believe Transport Packaging will start building a scriptable foundation for constructing robust solutions for staging workflows and pave the way for effective regression testing tools to be applied.

Quote from: redman at Sep 01, 2008, 10:53 AM

I’m wondering whether it’d be useful to create some documentation, best practices and tools like I’ve mentioned above to speed up development of sites in more formal environments, or whether people think each site/project is too unique to do this.

Absolutely; hopefully we can help define and identify the best practices and needed tools through this dialog.

if the client was integrated as a module directly into the manager, it would make it really easy for devs to create new documents on a production machine and then one with the same id in their local db, and it’d be secure too since devs wouldn’t need to contact the production site directly, and it would support multiple devs working simultaneously. also, coding this as a module would mean only people with this specific need would need to install the package, keeping the ui simple for normal users.

OG, what’s your reasoning for not wanting to use Phing? While it is another dependency, it allows svn tasks to be run and has a dbdeploy task. I haven’t started to use dbdeploy, but it looks like it could be useful for managing changes to the db schema.

The build process i’ve coded does the following:

development build:

* copies custom config files to the right place. The custom config files contain the db details. Different dbs (dev/staging/production) have different credentials for security, so the appropriate config file needs copying to the right place.
* purges the dev db
* imports the latest db snapshot
* imports snippets and chunks managed by svn into the db
* clears the modx site cache.

staging - this is run locally by devs after merging their changes into the branch, and prior to committing to the repo. I also intend for it to be run on the staging machine via CruiseControl following a trunk commit.

* purges the directory used for staging
* checks out each modx project that has been defined, builds each one (similar to the dev build process), removes .svn directories, and moves the files to the staging web root directory
* if it is running on the staging box, it will create documentation
* runs unit tests using phpunit (this part doesn’t work yet).

so, there’s quite a bit to do and phing integrates with all of the required tools.

Quote from: redman at Sep 02, 2008, 04:22 AM

OG, what’s your reasoning for not wanting to use Phing? While it is another dependency, it allows svn tasks to be run and has a dbdeploy task. I haven’t started to use dbdeploy, but it looks like it could be useful for managing changes to the db schema.

Don’t want that kind of dependency in a critical piece of the core, which I consider the Transport Packaging system. That said, it makes perfect sense in your custom environment. We use Ant directly to build the various distributions of MODx Revolution and will be doing to same with xPDO, so I have no aversion to the tools really, just want the Transport Packaging to work in systems where Phing/Ant is not a possibility. This spells Core Extension to me.

Also, note that the Transport Packaging does not use any SQL, but instead deals with database portable object serializations. This will be important for sharing packages across database platforms when we release MODx Revolution for PostgreSQL, SQLite, or other engines.

Quote from: redman at Sep 02, 2008, 04:22 AM

development build:

* copies custom config files to the right place. The custom config files contain the db details. Different dbs (dev/staging/production) have different credentials for security, so the appropriate config file needs copying to the right place.
* purges the dev db
* imports the latest db snapshot
* imports snippets and chunks managed by svn into the db
* clears the modx site cache.

staging - this is run locally by devs after merging their changes into the branch, and prior to committing to the repo. I also intend for it to be run on the staging machine via CruiseControl following a trunk commit.

* purges the directory used for staging
* checks out each modx project that has been defined, builds each one (similar to the dev build process), removes .svn directories, and moves the files to the staging web root directory
* if it is running on the staging box, it will create documentation
* runs unit tests using phpunit (this part doesn’t work yet).

so, there’s quite a bit to do and phing integrates with all of the required tools.

Perhaps this is getting a bit custom, but could definitely be crafted as a core extension (with Phing) to do these specific things, but the specific steps would need to be definable (again, maybe allowing custom Phing scripts to be executed would be the way). With Transport Packages, you could do many of the same things, plus have the power of smart objects being moved around and the MODx core could be scripted against, as well.

Ultimately, with Phing, I’d like to see the liberated MODx API be integrated using it’s own custom Tasks; then Phing itself could be used within MODx or outside of it to script these actions with the same intelligent data handling, even between servers.

ahh, maybe i didn’t make myself clear. the client/server app has no dependency on phing, so it could be used in the core, or as a core extension easily.

phing is used to build the site after a developer checks out a project, and to build the staging environment. Because not everyone would want this functionality, it’d probably be better to create some standalone tools to handle these things as part of a more complex configuration, in which case i think phing could be used for them.

I like the idea of the transport packages to remove dependence on mysql. is there any documentation on them?

In case anyone is interested, here is the client/server application i developed. It allows developers working on one machine to run a client that contacts a server running on another machine. The server contacts the production database and can create documents/chunks and snippets with the same IDs on the production and development databases.

The code is GPLd so other people can use it and modify it as they want.

Good feedback/contributions would be good

Thanks for sharing redman; we’ll certainly be checking it out.

Very interesting. I’m getting started on a new site, so I’ll set this up and use it and see how it goes.