Pushing Assets Folder - Staging to Production

Bottom Line:
I’m looking for your recommendation of the high performance way to migrate content changes to the "Assets" folder.

Background:
I’m a web designer whose client created 2 web sites using the services of a MODx CMS developer. They would like me to do their MySQL and Assets pushes for them.

I was advised (briefly) by the developer to:
1. Download Assets and subfolders from Staging
2. Delete Assets and subfolders from Production
3. Upload Staging’s Assets Folder and subfolders to Production
4. In PHPAdmin, export "site_" entries from Staging
5. In PHPAdmin, import "site_" entries from Staging to Production
6. Login to Production Manager and Clear Cache

Environment:
MODx Version 1.0.4, rev. 6981

My one web site (WEB1) has an Assets folder that weighs 344 MB.
My other web site (WEB2) has an Assets folder of 12 MB.

WEB1 on fast server with Telnet Available.
WEB2 on Network Solutions, slow and no Telnet, but which has an FTP File Manager which enables delete/copy/etc.

Problem: (Assets folder only)
WEB1 takes 11 minutes DL and 36 minutes UL, for 47 minutes
WEB2 takes 12 minutes DL and 17 minutes UL, for 29 minutes

Clearly these speeds are unacceptable and discourage updates. Also my invoicing of this time makes this very expensive for my client.

I just woke up to the fact that I should be doing a server copy from Staging to Production and quit this DL/UL nonsense, even at Comcast cable speed. Both Staging and Production are on same server. I have no idea at all why the developer didn’t encourage my client to do this, and in fact provide the procedures for them.

My Solution:
WEB1: Wrote a simple shell script that deletes Production Assets folder and copies over Staging Assets folder with permissions. Takes all of 15 seconds, instead of 47 minutes. Only pitfall is that the UNIX remove command fails with only the ’r’ switch because it wants to warn me about the hundreds of files that have ownership as "nobody". I researched this and chose not to go the suPHP route. Instead I force the removal without prompts by adding the ’f’ switch. The hosting company discouraged me from doing this but I’m doing it anyway and it seems to work fine.

WEB2: I’m using Network Solutions’ File Manager and manually deleting and copying the Assets folder. Response is almost instantaneous, instead of 29 minutes.

Question(s):
My Million$ question to all of you with experience updating MODx CMS Content is whether you agree with the path I’ve chosen? Do you see any dangers in my doing it this way? Is there a better way?

If you developed a MODx CMS site for a client, would you have covered the issue of pushing the Content, and in fact have provided a documented high speed solution suitable for the available server?

Thanks in advance to any of you that take the time to provide me with your advice/guidance/opinions.

Best ... Sam

Surprised nobody replied to this yet - I guess 4th July had something to do with it

I’m not really an expert on UNIX scripting so couldn’t say whether you’re putting things at risk or not. The only downside I see is that there is still a window (albeit significantly reduced) where your assets are not on the Production server. If you could find some way to route requests to an alternate location while the update is taking place then that would keep your site operating continuously. Perhaps you could duplicate your Production assets to this alternate location before making the upgrade - that way you’ll have a backup in case anything gets screwed up.

Other than that, it certainly seems a more efficient solution and economical for your client. The old way of downloading and re-uploading to the same server seems a bit silly to be honest, especially with those large file sizes. I think you’re less likely to have any data corruption resulting from connection problems this way too.

Thanks for writing, Lucas. Your comments are worth noting.

I’m pretty new to this solution but to avoid losing the Production assets folder in case of a mishap, I would likely rename it first using FTP or just adding that to my Unix script.

As for the site not being usable during the refresh of the assets folder, I could either 1) Do this off hours for the 2 minutes or so that service would be interrupted, or 2) Place the site offline for the refresh, which I would definitely have to do if the refresh took a hour or so.

BTW, I already learned the hard way that I could lose server permissions by deleting or renaming the Production assets folder and then copying over the Staging assets folders. Fortunately both the tools I’m using copy the permissions specified on the Staging server along with the files.

I’m glad you didn’t see anything wrong with my approach.

--- Sam

I think you’ll want to zip up your folders -- downloading assets individually ramps up the FTP time because for each file, the client has to validate.

Here’s what I would recommend:

Push your soon-to-be-live files to a special directory on the server, upload the database dump to a separate instance, then switch your httpd.conf directives so requests resolve to that new directory (the MODx conf file will point to the other db). Rebooting apache would enforce the switch, and there would never be a case where your requests would resolve to a halfway-incomplete site and you wouldn’t run the risk of losing your files via the delete scenario you outlined.

Once you have this set up, you can alternate between 2 different production directories: one will be the one that’s live, the other will be "on deck". Make sense?

Everett, I appreciate your support but frankly you’re way over my head. I’ve never heard of an httpd.conf file, have no idea right now how to zip and unzip files at a UNIX host, and I can’t follow the logic of uploading and downloading files. My intent by using server copies was to avoid uploads and downloads.

Before my brain explodes, I think I’m going to go with what I have right now. I appreciate another point of view and will review your ideas a bit further.

Thanks very much for writing ... Sam

We generally use rsync for this sort of operation. rsync is a handy tool that will synchronize directories and has a tonne of options to suit most applications. Because it uses diffs (so only the differences in a file are transferred, and not the whole file) and compression, it will be much much faster than your current method and basically, it’s designed to do exactly what you want - it even allows you to do a "dry run" so you can view the files that are going to be added/modified/deleted before actually executing the command. There are some GUIs available, but I’ve never used any of them myself so cannot comment on how good they are but as long as you’re semi comfortable with the command line, it’s quite trivial to use anyway. I won’t go into too much detail as there are many excellent tutorials around that explain it much better than I possibly could, but will give a brief outline on how you would use it so hopefully you can decide if it’s worth pursuing in your case.

The basic command would be something like:

rsync -arvuz -e ssh user@remotehost:/path/to/remote/assets/dir /path/to/local/assets/dir

In that command the options I’m passing are archive (keep ownership, permissions, symlinks etc), recursive, verbose, update (will not overwrite newer files on production) and compress (-arvuz). The "-e ssh" is just telling rsync to use SSH for the remote shell. If you wanted to perform a dry run you would also pass n (-arvuzn). There’s a bunch of other options like --delete to remove files on production that no longer exist on staging, --exclude to exclude any files or directories from syncing etc etc so it’s pretty versatile. In the past, we have even added a button in the manager that fires the command so users can deploy the assets themselves.

Happy to answer any questions/give you a hand if you do decide to give rsync a go.

Sam, sorry if I was over your head there, but this is definitely a case where a little bit of technical know-how will go a long way in saving you time (and save your clients money). A lot of the ideas are outlined here:

http://rtfm.modx.com/display/revolution20/Moving+Your+Site+to+a+New+Server

hk_modx, thanks for the rsynch tip. I plan to explore it today. If I go that route I likely will have a few questions for you. I appreciate your providing a sample command line with explanation.

Everett, thanks for that link. I haven’t installed MODx but I can use that information as good background.

At Butter (I work with hk_modx) we also use Git to do this for us.

Basically you would do this:

1. Commit new resources into Git on the production server and push them to your Git Repository
2. Pull the new resources from your Git repo onto your staging and do any merging you need to
3. Check Staging and once you are happy commit and push the merged repository back to the Git repo
4. Finally just pull the Git repo onto your production server again

You get the following benefits with this:

• Almost no downtime
• You can revert back if something goes wrong on production
• You can tag releases which will enable you to roll to any version you want in seconds

You could also use other source code control software however I would definitely recommend trying Git if you haven’t since it makes changes to files so much easier to manage.

Ed

Thanks for your attention to my posting, Ed. I’ll fire up Google and have a look at Git.