We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 36541
    • 222 Posts
    Recently, I’ve been thinking about a backup schema for my MODx sites. I use Subversion extensively, for both development and documents, so it is my preferred backup method. I want to achieve the following:


    • keep all site data in one repository,
    • keep site configuration, site templates, and site content in separate branches.

    The separation of site configuration and template/content is a must. Usually I have two instances of each site: 1) a production site on the production server, and 2) a development site on my local machine.

    Obviously, the differences in configuration between those two instances consist of paths, database credentials, etc. Apart from config files I also store the site configuration in SQL export from [tt]system_settings[/tt] table. This way I’m able to check out the config for the development or production site when I need it, without having to modify the template/content.

    Then I have two other branches in my SVN repository, which contain all templates and content related files. I separate templates from content by putting all media files related to template and design (images, js, etc.) into the template sub-folders and leaving those within [tt]/assets[/tt] for content related files. This way, I got three separate parts in the repository, which do not overwrite each-other:


    • site configuration,
    • site template,
    • site content.

    So, where’s the problem? Well, the problem is/are [tt]/assets/snippets[/tt] and [tt]/assets/plugins[/tt] folders. They contain elements that are related rather to the site logic, than its content or presentation. There are also other folders inside [tt]/assets[/tt] that belong to that category (or rather don’t belong to the content/presentation): [tt]cache, docs, export, import, site[/tt]. The problem is that the logic elements are mixed with content and presentation, and it complicates checking out with SVN repository.

    My proposition is to reorganize folder structure to reflect the purpose of their contents. I would roughly sketch the following structure:


    • [tt]/core[/tt]

      The MODx internals – logic, processors, etc. and site config. Now it is stored in /manager and /core (0.9.7).

    • [tt]/extensions[/tt]

      Snippets, plugins, and modules with their default templates (for example [tt]/assets/snippets/ditto/lang/language.inc.php[/tt]). Now it is stored in [tt]/assets/snippets[/tt], [tt]/assets/plugins[/tt] , and [tt]/assets/modules[/tt].

    • [tt]/templates[/tt]

      Site templates. Now they are stored in [tt]/assets/templates[/tt]. Template folder should also contain all overrides to snippets/plugins/modules templates if needed.

    • [tt]/content[/tt]

      All files which are related to the site content.

    Of course folder names could be different (and I’m for the ability to use custom names). I think this kind of folder structure is more logical and would make backing-up and porting a site much easier.

    What do you think?
      This is the web: the only thing you know about who will come is that you don't know who will come.
      • 30223
      • 1,010 Posts
      I’m certainly all for separating content related data from the template, script and configuration files. There’s another good reason to do so. It would make it much simpler to configure the File manager so that the average manager user could only fiddle with content related files.
        • 22303 MODX Staff
        • 10,725 Posts
        There are going to be three main directories, all configurable locations in the new core engine...

        /core (can go anywhere, even outside the web server document root)
        /assets (needs to be in document root)
        /manager (again, needs to be in document root)

        Each context can define a custom assets folder as well (i.e. manager/assets/) and this is only for presentation artifacts. As for add-ons, I’m considering where the best location for snippet logic is, and I believe it belongs in the core, separated distinctly from the presentation.

        In addition, there is going to be a mechanism for aggregating physical resources and data into installable packages with custom itineraries that describe their installation.
          • 36541
          • 222 Posts
          Quote from: OpenGeek at Jan 24, 2007, 10:47 AM

          Each context can define a custom assets folder as well (i.e. manager/assets/) and this is only for presentation artifacts.

          Does it mean that each context will reside in its own folder? That would make sense when combined with presentation templates and optional add-ons’ template files. But then, will a multi-language site with country specific subdomains (just for example) have to be split into separate contexts and, thus, folders? What if all subdomains will have exactly the same templating?

          Quote from: OpenGeek at Jan 24, 2007, 10:47 AM

          As for add-ons, I’m considering where the best location for snippet logic is, and I believe it belongs in the core, separated distinctly from the presentation.

          I agree that add-ons belong to the core. They are common for all contexts. But also I do remember that you put stress on scaling the core down to the bare engine and putting all non-critical functionality in addons. Following this logic it would be natural to separate the core and extensions to achieve a clean and easily maintainable architecture. So, I would go for separate folders as I described above.

          Quote from: OpenGeek at Jan 24, 2007, 10:47 AM

          In addition, there is going to be a mechanism for aggregating physical resources and data into installable packages with custom itineraries that describe their installation.

          Does it mean a way to distribute packages of files, chunks, snippets and tvs that form a solution? I think I recall you answered to that question already somewhere (probably multiple times) but want to get an explicit answer. If so, does it mean there will also be a way to work on chunks, tvs and snippets from outside manager? I mean, I would be more than happy if I could work just on physical files instead of database fields displayed in a textarea.
            This is the web: the only thing you know about who will come is that you don't know who will come.
            • 22303 MODX Staff
            • 10,725 Posts
            Quote from: grad at Jan 24, 2007, 01:24 PM

            Does it mean that each context will reside in its own folder? That would make sense when combined with presentation templates and optional add-ons’ template files. But then, will a multi-language site with country specific subdomains (just for example) have to be split into separate contexts and, thus, folders? What if all subdomains will have exactly the same templating?
            You can create a separate front controller (like the current index.php) for each context or you can have one controller handle requests to multiple contexts. It was designed specifically that way, and the idea is to always have the constants MODX_BASE_PATH, MODX_ASSETS_PATH, MODX_MANAGER_PATH, etc. available from any context, while the config values could be overridden by context when applicable... and components could take advantage of both local contextual values, as well as the ones for the ’controlling’ or ’primary’ context. I’m trying to make that as flexible as possible...

            Quote from: grad at Jan 24, 2007, 01:24 PM

            I agree that add-ons belong to the core. They are common for all contexts. But also I do remember that you put stress on scaling the core down to the bare engine and putting all non-critical functionality in addons. Following this logic it would be natural to separate the core and extensions to achieve a clean and easily maintainable architecture. So, I would go for separate folders as I described above.
            The real issue is separating the domain model from the views and controllers, and keeping them loosely coupled. This is why the piece about distributing physical packages of all these disparate file and db resources is so important IMO...

            Quote from: grad at Jan 24, 2007, 01:24 PM

            Does it mean a way to distribute packages of files, chunks, snippets and tvs that form a solution? I think I recall you answered to that question already somewhere (probably multiple times) but want to get an explicit answer. If so, does it mean there will also be a way to work on chunks, tvs and snippets from outside manager? I mean, I would be more than happy if I could work just on physical files instead of database fields displayed in a textarea.
            Well, no, these are separate issues; the packages would store the physical files and their dependencies to other files and/or database records, along with an intinerary for installation on a target. The database records, to avoid MySQL platform dependencies, will be stored as PHP code that, when executed, rebuild the object instances in the deployment it is installed/imported into. These are then saved to the new db via the xPDO scaffolding according to the itinerary (which can be a chain of PHP scripts and/or interactive events that present UI in the installer). There would also be a package building interface for constructing the packages.

            As for file-system access, I plan to eventually enable protocol-based access (i.e. WebDAV/DeltaV, FTP, etc.) to the database-managed content, though I am also working on a way to optionally store the content for all the element types (snippets, chunks, templates, etc.) on the file system directly, as opposed to storing it in the content table itself. But this would likely be as a copy of the current revision constructed from the database, and I have not yet solved how writes directly to those files on the filesystem could be detected by MODx so it could create a new revision, etc. Could definitely be done, but I think protocol access to the database managed content is the better option in the long run.

            The alternative approach for file-system access, would be to use MODx as a framework only, and create individual PHP files that bootstrap the MODx class and use only the parts of the framework they need, per file. I plan to use this in conjunction with context support to make it much easier to use MODx within existing PHP scripts that are not built in a MVC paradigm (e.g. MODx menus in WordPress).

            Remember the more feedback the better on this, just to make sure these ideas are the right way to go. So feel free to pick what I’m saying apart... smiley
              • 36541
              • 222 Posts
              Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

              the idea is to always have the constants MODX_BASE_PATH, MODX_ASSETS_PATH, MODX_MANAGER_PATH, etc. available from any context, while the config values could be overridden by context when applicable... and components could take advantage of both local contextual values, as well as the ones for the ’controlling’ or ’primary’ context.

              That’s what I meant by local override of default config.

              Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

              the packages would store the physical files and their dependencies to other files and/or database records, along with an intinerary for installation on a target. (..) There would also be a package building interface for constructing the packages.

              This means an easy way to distribute add-ons and even custom solutions with sample data. I think it will have a really warm welcome. I think it could be a good way to move a site from development to production environment.

              Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

              As for file-system access, I plan to eventually enable protocol-based access (i.e. WebDAV/DeltaV, FTP, etc.) to the database-managed content,

              That would be very useful if only access to the share would not be limited to a narrow choice of IDEs (Eclipse and the like). If the share would be mountable using standard OS tools, than it will be OK. But then would revisioning be handled natively through it?

              Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

              I am also working on a way to optionally store the content for all the element types (snippets, chunks, templates, etc.) on the file system directly, as opposed to storing it in the content table itself. But this would likely be as a copy of the current revision constructed from the database, and I have not yet solved how writes directly to those files on the filesystem could be detected by MODx so it could create a new revision, etc. Could definitely be done, but I think protocol access to the database managed content is the better option in the long run.

              Assuming the use of database storage, a protocol access would really be a better option. One could always copy the file to a local working folder and do with them whatever is needed. However, I’m pretty sure that once this option is available users will start to try to mimic (or even switch) to filesystem storage.

              Now, that would be a real pro for MODx if there was a choice between database based and pure filesystem based storage. The discussion about database vs. filesystem in not new and, as it’s often the case with such subjects, it’s hard to convince each other. But filesystem storage has many advantages over database one. Some good points and discussion on it can be found on PmWiki Flat File Advantages article.

              Anyway, I think that if only it is possible to provide a choice between database storage and pure filesystem storage, then MODx should do it. That would be another great step in direction of flexibility. And I would love to use it and combine it with SVN revisioning. laugh

              Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

              The alternative approach for file-system access, would be to use MODx as a framework only, and create individual PHP files that bootstrap the MODx class and use only the parts of the framework they need, per file. I plan to use this in conjunction with context support to make it much easier to use MODx within existing PHP scripts that are not built in a MVC paradigm (e.g. MODx menus in WordPress).

              That goes beyond my current understanding of web applications architecture. I’m just a designer with some programming skills and such highly abstract concepts are hard for me to wrap my head around them.

              Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

              Remember the more feedback the better on this, just to make sure these ideas are the right way to go. So feel free to pick what I’m saying apart... smiley

              I think I succeeded wink
                This is the web: the only thing you know about who will come is that you don't know who will come.
                • 27376
                • 576 Posts
                Quote from: OpenGeek at Jan 24, 2007, 10:47 AM

                /core (can go anywhere, even outside the web server document root)

                As for add-ons, I’m considering where the best location for snippet logic is, and I believe it belongs in the core, separated distinctly from the presentation.
                If you put the add-ons (i’m assuming snippets, modules, etc) in the core, then some snippets and modules might not function properly if the core is placed outside the document root because AjaxSearch for instance, includes javascript which is included in pages.

                I for one would take advantage of placing the core outside the doc root because my server will soon be powering multiple MODx sites and having all the sites work off the same core would make for easier updating and save space.

                So I propose that if the add-ons are not to remain in the assets/ directory, it should be placed in it’s own directory (like ’add-ons/’) which is user-defineable.
                Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

                I am also working on a way to optionally store the content for all the element types (snippets, chunks, templates, etc.) on the file system directly, as opposed to storing it in the content table itself.
                Question: After only briefly looking over the 0.9.7 architecture I may not know what I’m talking about yet. But would it be possible to create a new xPDO class that, instead of making tables in the database, it would make files on the system? I imagine you’ve already considered that, but just thought I’d bring it in to the thread smiley
                xPDO looks very cool btw!
                  • 22303 MODX Staff
                  • 10,725 Posts
                  Quote from: sirlancelot at Feb 14, 2007, 11:44 AM

                  So I propose that if the add-ons are not to remain in the assets/ directory, it should be placed in it’s own directory (like ’add-ons/’) which is user-defineable.
                  That’s what the context-specific assets directories are for: containing all the web resources that would have to be served from directories accessible by the web server. It would be the responsibility of the import/export mechanisms to a) define the specific location within an assets directory it would live and b) decide if the web resource would be imported into the ’main’ assets directory (defined by MODX_ASSETS_PATH), or allow the $modx->config[’assets_path’] to define it’s root location, which can be overridden by context (i.e. if I wanted to import a component only into a specific context such as the ’mgr’ context).

                  Quote from: sirlancelot at Feb 14, 2007, 11:44 AM

                  Quote from: OpenGeek at Jan 24, 2007, 02:15 PM

                  I am also working on a way to optionally store the content for all the element types (snippets, chunks, templates, etc.) on the file system directly, as opposed to storing it in the content table itself.
                  Question: After only briefly looking over the 0.9.7 architecture I may not know what I’m talking about yet. But would it be possible to create a new xPDO class that, instead of making tables in the database, it would make files on the system? I imagine you’ve already considered that, but just thought I’d bring it in to the thread smiley
                  xPDO looks very cool btw!
                  Well, the file caching mechanisms in xPDO and the MODx extension of xPDO (yes, class modX extends xPDO) already write executable PHP code to file. This is used in writing the script elements (modSnippet, modPlugin, modModule) to file (as functions) to be included, rather than having the source loaded in memory for all scripts, and being eval()’d each time. It’s also used for the database result set caching, which stores the entire PHP object, including content, as a PHP file in the cache.

                  It will be fairly trivial to write other types of content to file as well. The challenge in my mind, is to create a consistent way to allow these resources to be constructed from database content, be edited and managed directly as the file, and then get those changes reflected back into the database. Then we can have WebDAV, SVN, FTP, or other protocol access directly to the content elements in MODx without sacrificing the benefits of keeping the actual content ’managed’ and eventually, part of the internal revisioning system. But, we could get there incrementally, and in the meantime could accomplish similar things by simply providing extensions of the various classes that only look for their source content on the file system.
                    • 27376
                    • 576 Posts
                    Quote from: OpenGeek at Feb 14, 2007, 12:08 PM

                    That’s what the context-specific assets directories are for: containing all the web resources that would have to be served from directories accessible by the web server.
                    Ah, that part confused me..... So let me get this straight: There is a base (core shall we say) assets directory where core snippets, modules are stored, and then on top of that will be an additional assets directory where more snippets can be stored? Or is there just one?
                      • 22303 MODX Staff
                      • 10,725 Posts
                      Quote from: sirlancelot at Feb 14, 2007, 03:51 PM

                      Ah, that part confused me..... So let me get this straight: There is a base (core shall we say) assets directory where core snippets, modules are stored, and then on top of that will be an additional assets directory where more snippets can be stored? Or is there just one?
                      Typically, it would be one, shared by all contexts, but the ability to override that by context, and isolate some physical web resources to a particular context, might come in handy...