Conceptual structure for versioning

I’ve read through the many posts requesting or responding to requests for versioning, butI haven’t yet seen a useful description -- in technical terms -- of how this is going to be accomplished. All I know at this point is that it is coming in a future version, that it will probably contain just the changed elements (rather than a duplicate of everything), that there may be an attempt to combine internationalization and versioning, and that it may not be attached to documents per se, but rather to snippets and the other items that create the content. That’s a summary of what I’ve read, but it’s kind of hard to plan for the future changes without having a few more specifics. I realize that the specifics may not be decided yet, but if that’s the case, I’d at least like to hear the general direction that the developers are currently leaning. So here are the specific questions that would help me if I had answers:

1. Are the various versions going to be A) kept in the same table as the "official" live version, but differentiated by some sort of flag, such as a field that says "official" or "approved" or something like that... OR are the previous versions going to be B) kept in a different table, such as a table specifically for previous versions ... OR is there C) some other method?

2. If the first case (A) is true, what will be the marker used?

3. If the second case (B) is true, would there be a duplicate table for all relevant tables (i.e. those that would need versioning), or would there be a single table to hold all of the previous versions?

4. If the last case (C) is true, could you provide at least a hint of what that might look like in terms of the database structure?

5. If you have decided to merge the ideas of internationalization and versioning, will the international versions meet condition (A), (B), or (C)?

6. Have you thought of going one step further and combining a) internationization, b) versioning, and c) other miscellaneous types of variations on the "original"? The other miscellaneous types of variations might include: a) different reading levels b) content customized according to a target audience, c) user-defined customizations, etc. It would be handy to be able to define our own kinds of "variations" along these lines... and along lines that I haven’t thought of yet. We could have several official versions, for example: a) English, default reading level, b) English, simplified reading level, c) English, default reading level with user customizations, d) English, simplified reading level with user customizations, e) French, default reading level, f) French, simplified reading level, and so on. Yes, it could get messy, and no, I don’t have specific plans at this moment to do something quite like that, but I have ideas for the future that would involve something like that. By providing a generic "variation architecture", it would allow me to define those variations later, according to the needs of my future projects.

Thanks for your input and feedback.

The answer is C) some other method, which I’ll try to describe, in easily digestible bullet points and as concisely as I can.
[*] System Settings defines configuration defaults, including default cultural settings (languages, date/currency formatting, etc.)
[*] Contexts can then override system settings for distinct domains, subdomains, site subsections, localized sections, or even programmatically; use this to divide the site into manageable pieces, or to customize based on other contextual data.
[*] Resources (aka documents or pages) can aggregate content Elements in just about any way imaginable.
[*] Elements have source content and produce content for output or consumption by other Elements, and can point to specific source Content based on Context and/or Culture.
[*] Content is unique from Elements and an Element can point to various Content records based on Culture, Context, or directly via the MODx API.
[*] Revisions to Content are stored incrementally only; the baseline Revision is the only full copy.
[*] Rollback of Revisions consists of reconstructing a specific Revision by patching from the baseline with the relevant incremental diffs.
[*] Archiving of Revisions will be possible by exporting up to a specific Revision, and resetting the baseline at the next Revision. Where and how it is exported will be completely independent and configurable.

Using this structure I believe we can support l10n (localization) and i18n (internationalization) efforts easily in MODx, as well as achieve just about any other kind of contextual customizations/variations of content.

Also worth mentioning regarding internationalization, I’m working on something I call a Lexicon, which will be like a dictionary of common words/phrases that can be used for anything from internationalizing a page similarly to the way $_lang[’blah’] is used now, to auto-linking, to glossary generation, to just about any other kind of lexical analysis.

BTW, just for reference from http://en.wikipedia.org/wiki/Internationalization_and_localization ...

The distinction between internationalization and localization is subtle but important. Internationalization is the adaptation of products for potential use virtually everywhere, while localization is the addition of special features for use in a specific locale. The processes are complementary, and must be combined to lead to the objective of a system that works globally.

Thanks for the explanation. This solidified a few things in my mind, though not all. The explanation probably is about as clear as it can be without me actually seeing the database structure. I don’t suppose there’s any way that I could be sent either a database schema or an SQL query to set up the database (e.g. like the ones which can be generated by phpmyadmin’s interface and saved as a zip file)? I mean just the database. Not the MODx scripts or anything. Here’s my reason for asking:

I’m using MODx as the basic framework for a college web site. I’ve got a team of people creating snippets, scripts and modules that access a database that I designed which keeps track of people (faculty, staff, etc.), courses, academic units, programs, initiatives, and a lot of other stuff. So far so good. It works beautifully, but I am required to create a versioning system for the college database, and I must also include a workflow management solution that saves drafts (not of pages, but of data within my custom database).

What I want to do is create a system that at least approximates the functionality of future versions of MODx so that I can end up with an integrated whole, where my versions and drafts can communicate with the versions and drafts of pages (or perhaps "resources" to use your terminology). I need to accommodate drafts not only of new database entries but edits of existing entries as well, putting them "on hold" until approval by a designated approver (and I know that the workflow issues are not on the slate for version 1.0, but I plan on using a modified version-tracking system to help me with that). Our scripts are writing to both the MODx database and our custom college database, and I just want the system as a whole to work as seamlessly as possible.

Being able to see the database setup would give me the technical ability to at least come close to the system anticipated for future versions of MODx. I’m fully aware that MODx is a work in progress and that things could change, possibly even drastically. I’m ok with that. It’s just that I’ve been laboring over this design issue for a while, inventing the wheel, when I know that you’ve got something over there that is at least round and rollable, even if it’s not the "Wheel 1.0" that we’ve all been waiting for.

Even if it’s off list, sent to me privately, and if it requires signing a non-disclosure agreement. I’m amenable to any of those things.

Oh, and by the way, I will say that I did a LOT of research before choosing MODx, and I’m committed to it, mainly because it is so flexible and almost infinitely modular... but I’m at a point where my design decisions could have long-range repercussions if later I have to backtrack and redesign the versioning system, if it ends up being incompatible with the future MODx system.

... or perhaps I could ask my question this way:

In which table will the revisions be stored? Will the revisions be an extra field added on to the database where the originals are kept? For example, I could envision a design where there are just one or two fields used to keep track of revisions, and these fields are a part of the same record that stores the "official" or default version. These one or two fields could be parsed to reconstruct previous versions of the page, as well as alternate versions. Perhaps the field contains an XML-formatted document, which catalogs the different versions using XML tags as markers, or something like that. Or if not XML, it would at least contain an easily-parsed text structure that codes the differences. ... but the root of my question is where the data is stored within the database.

I feel like I’m running the risk of being pesky with my questions, and I hope my comments aren’t taken that way. I’m just hoping to be able to plan my project as well as I can.

Not taken as pesky, but at the same time, I’m not ready to reveal my final data structures to the world. This is for a number of reasons, but the important one is that exposing those details at this point could allow someone else to capitalize on my efforts before I can.

In any case, the versioning system I have in mind treats revisions as a separate entity/table of GNU diff content, and the current "version" of a piece of Content is stored whole on the intersection of that specific Content record and a Culture record; only the incremental diffs to a specific Content record are stored as records in the Revision table, after the baseline or base revision.

Depending on the amount of content you plan to manage, it may be useful to create Content and Revision tables specific to certain objects in your data model though. Just keep in mind there are always balances to keep in the trade-offs of using generic storage systems vs. highly customized/optimized ones, like choosing whether to use one table for Revisions or one table for each type of Revision.

A curious newbie (who’s recently jumped into modx fully from webgui and loving it ) wondering general roadmap status of versioning?

Also, are modX developers open to (or need any) any usability input with implementing new features such as versioning?

I think you’ll find us very open to solid ideas from non-core team members, especially if they’re accompanied by code! In fact it’s frequently helpful community members doing stuff like this and helping out that find themselves with team membership invitations.

Please consider the below versioning strategy, which will yield the following benefits:

1. Quick, relatively easy, yet reliable implementation of versioning.
2. Easy for authorized users to browse and diff the versions, via some nice GUIs on many platforms.
3. Authorized users can use any external editor, on any platform, to update MODx pages.

Here is the strategy:

- For each object (i.e., page, template, snippet, chunk, plugin, etc.) that is versioning-enabled, on each update, the object is checked into (added or committed) into a predetermined Subversion repository.

- When editing an object, a button is available called ’checkout’, accompanied by a box that accepts a revision number or a symbol such as * which means latest. When the user activates this button, the specified revision is obtained from the Subversion repository if possible and replaces the current contents.

That’s it.

Authorized users can separately use their preferred Subversion client to browse, diff, edit the Subversion repository contents.

Authorized users can optionally use their own favorite text/html/php editor to edit files checked out of the Subversion repository, and then (as above) tell MODx to check out from Subversion into the database. The time lag between the manual update into Subversion and the check-out into MODx gives us a poor man’s implementation of one level of staging.

Users not authorized to directly access the Subversion repository can still roll back to any old version. A diff facility for them can be addd later, in due time.

Rahul

That’s an interesting idea indeed Rahul, and quite powerful in it’s simplicity. However, everyone may not have Subversion capabilities on their server. What’s your suggestion in that instance?

Suppose the implementation is kept simple. Track object contents linearly. No branching, no tagging.

Can use object id only, possibly with a hashing scheme (i.e., store object 10000 as "10/00/10000") if needed to prevent back-end thrashing on large systems.

So now we have a simple, pluggable, back-end API for versioning. Given an object id, (a) check to see if it exists, (b) get its list of konwn revision numbers, (c) retrieve a specified revision number, (d) add a new object; (e) commit a new revision.

The release notes can then say, "If you have Subversion, revisioning is available today. Otherwise, just wait a little -- the hooks are all in place and we will add a built-in versioning back-end later."

OPTIONALLY:

Require every web page to have an alias, and use that alias (converted to a pathname based on the page’s position in the web site hierarchy) as the object name in the revision-control back-end. A change in the alias of a page, or the name of a chunk/template/plugin etc., causes a back-end rename operation. This increases the complexity of the API only very slightly. But now our repository nicely reflects the hierarchical structure of the web site.

ALSO:

Always store some basic object info, e.g. object id, in the first line sent to the revision-control back-end. This line is stripped when checking out the object. Then if somebody accesses the repository directly and renames an object, MODx could in theory find out where the object went, with an exhaustive search, in a maintenance operation. This first line can be expanded later to track more object properties, such as published/unpublished, in menu/not in menu.

Rahul

P.S. Could use CVS or RCS -- more universally available on typical Linux server installs used by web-hosting providers. Not so nice for binary objects or for remote browsing. Renames are still easy -- just bypass the CVS or RCS interface and physically move the back-end object.

-- Edited to add: (e) commit a new revision..