Core Extension to manage everything from filesystem (*OR* manager UI)

According to MODx developers, Revolution 2.1 database will have "a complete set of Audit fields on all transactional tables, including createdon, editedon, and deletedon time stamps, as well as createdby, editedby, and deletedby ownership."

This opens the door to a big new possibility: gaining the option to manage pretty everything (i.e. resources and elements) from plain text files.
That means that you can use your beloved IDE and forget copy/paste into the manager UI.
That means that you can use version control on everything.
That means that you can sync production and development servers just synching two directories on filesystem, no more note down changes done during the day and patiently copy/paste them from the manager UI.

All of this without renouncing to the use of the manager UI whenever you want to.
The two worlds can co-exist interchangeably, without even moving a switch: an automatic synching is possible.

With those timestamps jumping in v2.1, it should be feasible a Core Extension able to transparently manage, with some conventions, bidirectional synch between the database and an ad-hoc Structure in the FileSystem (let’s abbreviate it "SFS"), where files contain all the informations required to populate the corresponding records in the MODx database.

Let’s speak here about this.

To understand the subject, it’s a good idea you need to read the following posts (they’re all in the same thread):

stephenrs: read this first
insert_nick: ...then the proposal...
OpenGeek: ...then the dev...
OpenGeek: ...reasoning

While I’m here, I think opening a poll to measure if there’s interest won’t hurt anyone, so vote for it if you think such a Core Extension would meet your needs.

And again, this can be done before 2.1 by using the MODx Registry (a type of simple message queue that can contain executable PHP messages with logic or data) to log changes to any database objects (even custom database objects). This is without having the new audit fields in place, basically by setting callback functions on save and remove actions, and logging the changes as messages in the queue, which can then be dealt with using Plugins (and eventually a custom UI) on the manager side. In fact, forms of native versioning (also planned for Resource/Element content in 2.1) can also be achieved through this Registry, and I’ll work on some examples of the bare necessities for getting these things started over the next few days.

However, I still think the Package Management system could be utilized as a foundation to handle promotion and deployment issues that simply cannot be managed with traditional version control tools, with or without a file-to-db/db-to-file synchronization solution. IMO, these version control tools are great, but for the development side of the house only, for managing source code development and the construction of packages to deploy changes from a development environment to a target environment. Logging all the changes made in one deployment and synchronizing them with a production environment via Subversion (for example) will not work in many scenarios, either because during development you tend to "try out" a lot of things that will not go into the end product, and/or because production data is being changed directly in production while development is going on. I think being able to manage promotion of specific changes though a combination of Registry-based audit logs and Transport Packages, which can contain scripted logic where needed to validate dependencies or resolve conflicts, is going to be the best solution for the widest number of scenarios. And, we can start the work of making this manageable without developer intervention after we identify and work through the specifics of the solution as developers.

IOW, we have multiple life-cycles to consider managing here, Development life-cycles, Production Change life-cycles, and potentially Staging Environment life-cycles as well. I look forward to exploring these in more depth this week with those interested in the discussion and the challenges presented by the proposed solutions.

though a combination of Registry-based audit logs and Transport Packages..

OK opengeek, to get the ball rolling on this area :-

The hard part here to my mind is versioning, most users need the ability to roll back easily, either individual entities or the whole database to a snapshot in time from the past, akin to an svn revision if you like. Some users just need to know who changed what and when and what they changed but won’t need to do anything else.

The audit log will tell you who changed what and when but you still need to store the changes themselves with the log somehow. If you can do this you can synch up a production db say with a dev one by ’replaying’ the changes into it, rather like an svn merge, where in case of conflict the user can make an ’accept yours/mine’ decision. Don’t forget you have to version everything, resources often depend on chunks/snippets/TV’s etc. Also what packages were installed/updated on the dev system and at what time? Do we ’replay’ these or assume the user will pre-install these before he starts?. The big test of course is rollback, say a user synced a dev box to a prod box and completely borked it, how do we recover? One way to do it of course is to be transactional and only ’commit’ when you are happy, a la oracle say, but hard.

This can be a very complex area, I suggest first we come up with a set of requirements of what we are trying to achieve, especially initially, and start from there.

Hi insert_nick, a few of your points :-

This opens the door to a big new possibility: gaining the option to manage pretty everything (i.e. resources and elements) from plain text files.
That means that you can use your beloved IDE and forget copy/paste into the manager UI.
That means that you can use version control on everything.
That means that you can sync production and development servers just synching two directories on filesystem, no more note down changes done during the day and patiently copy/paste them from the manager UI.

You can do this without the versioning at the mo if you use the modxfs approach(c based or php based). I believe this is useful for dev houses but possibly not much for the average user. This gives you no real audit trail though.

This may be a dumb idea. And if so, let me know why.

(Of course this plugin would only work if you had access to git)
What if each table and record had a file in the database like:
/Content/
/Content/1
/Content/2
/TV/
/TV1/

On any doc manipulation, you save the INSERT sql for that table in that file, then update the git repo.
That handles the versioning and cuts out a huge chunk of programming time. The rest would be handling the management of users.

The main reason to do this for stage 1 would be it’s quicker to use a VersionControl system that already exists than rebuild one.

I know it’s kind of crazy. But it would be efficient. And seems like good technical debt for an alpha project.

PS - I got the idea from someone using Trac as their versioning system. And this is a link to someone who did this as a cronjob. Not as robust as my idea: http://www.viget.com/extend/backup-your-database-in-git/

Hi ChucTrukk,

One prob here is keeping the two in sync, if I create/delete a db entity through the manager you would have to reflect this into git, you could of course trap all such occurrences and do this.

Presumably((if I’m understanding you correctly here), if you wanted to roll back an entity you would have to find it in git, extract it and paste it back into the db through the manager(this would of course now reflect itself back into git). This is clumsy if you are a ’normal’ user using the manager, unless you write a GUI to do this. Versioning is only really of any use if at some point to you intend to re-instate data that you’ve lost, so this must be easy also.

Also the implementation detail may become more comples than first thought if you deleted an entity would you delete it from git or ’mark ’ it as deleted?
The db can have resources with the same names cos they have different id’s, git may not like files being named the same, so it may end up with the id being mangled into the filename etc.


Agreed initially its easier to interface to an already built versioning system, the prob as ever is the interface to and its associated workflow.

I already tried to push for something in this direction about one month and one year ago, and good shamblett made the step of modifying his Auditor accordingly, but no devs succeeded (or engaged) in doing a usable component for what I asked. So there was a stale.

Now I don’t know MODx Registry but I imagine it’s something that gets us close to where we already were with shamblett’s upgraded Auditor, with the same issues that we discussed at the time regarding the db synch facilities as long as we start speaking about rollbacks, undo/redo, and such.

So what I am proposing this time is a bit different in concept, and a bit simpler too; as I’m a little scared of a new stale, let me clarify it better.

Of course the Extension could evolve in time with something bigger, with undo/redo facilities, snapshots, etc. But they will eventually be a plus, not the core.
So let me focus in explanation on the core concept:

In the manager UI, I see the core Extension exposing me only one new switch (again, with time this number could grow with other features/options, but let’s focus):

[ ] enable sfs rapid development mode (?)

...unselected, in default installation, to use MODx like we all do it now, writing (or pasting) stuff in the manager UI, by clicking edit button, save, and so on.

If the switch is selected by the user, essentially three things have to happen:

1) when the user changes switch value, if it does not already exist, it creates the sfs: a collection of files structured in folders, which are "twins" of documents and elements found in db, or - generally speaking - whatever is editable from the manager UI; if the sfs already exist, it alerts that a sfs already exists, and asks in which direction it has to do a clone: db->sfs, or sfs->db? It gives an hint saying that the freshest thing has been found in the db, or in the sfs; are you sure? type SURE all uppercase if you are SURE; ok it seems you are sure, let’s go. No rollbacks; no undo/redos; not at all;

2) when the sfs-RDM mode is on, there’s no way to delete stuff from the manager UI: when you try a popup appears explaining that sfs rapid development mode is enabled, so to delete stuff you have to delete the twin file; an option to this could be that "deleting" is interpreted as a "mark for deletion" when sfs-RDM is enabled, so that the Core Extension will do the real dirty work by killing the thing both in db and in sfs (but this probably would require a new "markdelete" field in modx tables);

3) when the sfs-RDM mode is on, when a page of the MODx-driven site/app or of the MODx manager is loaded, an automatic synch process between the db and the sfs is done by the Core Extension; the rule is that the freshest twin wins, and clones itself on its counterpart.

I can’t easily see how this can be done with the MODx Registry, but maybe it’s me. With the Audit fields on all the transactional tables I can see it well.

Please note that in my reasoning, every other thing besides those three, from editing stuff to versioning management, ftp synching, multiple developers, development/production environments, whatever, is left to user’s personal preferences, tastes, requirements, tools.

Quote from: shamblett at Oct 19, 2009, 11:36 AM

Hi ChucTrukk,

1. One prob here is keeping the two in sync, if I create/delete a db entity through the manager you would have to reflect this into git, you could of course trap all such occurrences and do this.

2. Presumably((if I’m understanding you correctly here), if you wanted to roll back an entity you would have to find it in git, extract it and paste it back into the db through the manager(this would of course now reflect itself back into git). This is clumsy if you are a ’normal’ user using the manager, unless you write a GUI to do this. Versioning is only really of any use if at some point to you intend to re-instate data that you’ve lost, so this must be easy also.

3. Also the implementation detail may become more comples than first thought if you deleted an entity would you delete it from git or ’mark ’ it as deleted?

4. The db can have resources with the same names cos they have different id’s, git may not like files being named the same, so it may end up with the id being mangled into the filename etc.


Agreed initially its easier to interface to an already built versioning system, the prob as ever is the interface to and its associated workflow.

1. We could use a modx plugin to say onSave, onDelete, etc. write the current info to the repo and commit

2. I do envision a GUI to see revisions and changes.
a) The user would: look through revisions (similar to browse through the revisions and click restore for the revision chosen
b) Git would create a new revision with the resources from the revised version and commit the changes. Then update the db with the revised changes.

3) Git would delete the resource and commit if it’s deleted. The revision history will kep track of deletion for restoration

4) Each resource in the git repository would be named by their primary key(2). That way their could be no collision.

Quote from: shamblett at Oct 19, 2009, 11:36 AM

Agreed initially its easier to interface to an already built versioning system, the prob as ever is the interface to and its associated workflow.

To enforce my point on what I’ve written above, i.e. letting user manage stuff with methods and tools he likes/needs and focus on a pure automatic just-synch-and-shutup db-sfs feature in MODx through a Core Extension or whatever else, I want to highlight that in my proposal the twin files in sfs contain a sort of yml part before the "real" content, so problems like the one that shamblett arises about filenames simply do not touch us with my approach.

And yes, if your favourite tool is git, you get git.
If your favourite tool is [put_name_here], you get it.

contain a sort of yml part

,

Yes, the meta-data associated with an entity as held by the database.

The more database meta-data you expose in whatever form whether it be filesystem attributes in the form of setattr/getattr or YML markup the more your filesystem representation looks like a MODx database ’outside’ of a MODx database. You will need more than just documents and elements, taking just snippets for example you would need to store any associated property sets that configure the snippet, if you didn’t the snippet wouldn’t behave properly when applied. Then there’s users, system settings, packages etc.

This can of course be done and managed from within the manager but eventually I think you’ll ask the question ’seeing as we are using essentially a MODx database structure why not just use one and be done?’ This leads back to using something like Auditor, that neatly versions every single database update with all meta-data intact but not in the vcs of your choice and still leaves you with a GUI to write to make sense of it all.

I think there are several goals here, we have :-

1. Auditing - who did what and when.
2. Versioning - keep this version I may want it later.
3. Database synching - make this database look like this one quickly and safely
4. Rapid development - I want to edit an entity in my editor of choice outside the manager and see its effects instantly when I save it.

and PS I may want to use some combination of my own workflow/toolset to achieve some/all/any of the above from within the manager.

Lots of food for thought here.

I get your point, but have to disagree: I don’t urge to replicate the MODx database on the filesystem, but to have the means of doing the development on a project without having to copy/paste code in form fields.

Staying with the snippet example you mention, options that are not found in the sort-of-yml metadata on sfs won’t be changed. If you need change, you have to explicitly add the property in the file on sfs (and the extension will do the db update for you), or just change it from the manager UI. The king will always be the manager UI of course, but I’m speaking of offering to the developer a way to just load the text editor, make changes on file, and then CTRL-S, Alt-TAB to browser, F5.
End of day, can’t remember exactly what files I changed, or I have no time to re-do the work on the remote installation; oh how easy with the sfs-RDM switch enabled also on the remote site, especially in initial stage when development is hot: I’ll just upload the whole snippets sfs folder. And I have no need to visit the remote site/app or manager, because the extension, as soon as someone visits the site/app or the manager, detects that sfs is more fresh than db for some snippets, and triggers update for them on the db accordingly.

And support for stuff other than documents and elements is not necessary, because the "real" development, the one that involves daily copy/paste, is at least 80% done on them. Other thing (users, etc.) could be considered for the future, but imho it’s better to focus on what’s of more usage during development.

The extension in my mind has to be thought as an aid for hot development, not as an alternative to the manager UI, or as an imposed swiss knife for all kind of needs. Everyone has his own preferred swiss knife, so what I’m proposing is an hook on the belt where to attach it.