We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 12983
    • 108 Posts
    *DISCLAIMER: the sfs-RDM switch on remote/production site should be off during not-hot development periods, because it will add some time on each page render, as it needs at least to query the database and the filesystem in order to detect if there are any differences in timestamps. If the project is not under heavy development, the extension does nothing than that, but the time needed for the check could maybe reach 1 second for very big projects (can anyone elaborate some rough estimate?).
      • 34017
      • 898 Posts
      Quote from: shamblett at Oct 19, 2009, 01:23 PM

      1. Auditing - who did what and when.
      2. Versioning - keep this version I may want it later.
      3. Database synching - make this database look like this one quickly and safely
      4. Rapid development - I want to edit an entity in my editor of choice outside the manager and see its effects instantly when I save it.

      I think the Rapid Development doesn’t belong with that list. It is a seperate thing. Auditing can versioning do go together. I just wanted to mention my thoughts were never to be able to edit the FS, and just have modx read that as the db. It was just using the FS as the input for Git.

      And I do think each row being a file named with their PK, and each field being represented via YAML would work really nicely.
        Chuck the Trukk
        ProWebscape.com :: Nashville-WebDesign.com
        - - - - - - - -
        What are TV's? Here's some info below.
        http://modxcms.com/forums/index.php/topic,21081.msg159009.html#msg1590091
        http://modxcms.com/forums/index.php/topic,14957.msg97008.html#msg97008
        • 12983
        • 108 Posts
        Btw this thread has been specifically open regarding the rapid development purpose: see the first link I pointed to in the first post to help understanding the subject (which I understand is not immediately understandable - sorry for the word play wink

        Quote from: insert_nick at Oct 18, 2009, 09:58 PM
        To understand the subject, it’s a good idea to read the following posts (they’re all in the same thread):

        stephenrs: read this first
        insert_nick: ...then the proposal...
        OpenGeek: ...then the dev...
        OpenGeek: ...reasoning

        EDIT: modified "it’s a good idea" in "you need".


        In other words, points 3. and 4. in shamblett’s list:

        Quote from: shamblett at Oct 19, 2009, 01:23 PM

        3. Database synching - make this database look like this one quickly and safely
        4. Rapid development - I want to edit an entity in my editor of choice outside the manager and see its effects instantly when I save it.


        Regarding points 1. and 2., which also do interest me very much (and I think someone else too), I suggest to open a separate thread.
          • 26903
          • 1,336 Posts
          Yes, my list above was quite global, each of the items can support the others but don’t need to.

          insert_nick, see where your coming from now from the rapid dev approach, you can of course do this without worrying about the other things in my list, I was trying to edge the discussion towards a more comprehensive solution to all this whereby we could update the core say to support all of these functions. We would still need 3PC support but at least the basics would be there.

          Separate threads can be used for each of the points so we can at least get everyone’s requirements and go from there.
            Use MODx, or the cat gets it!
            • 12983
            • 108 Posts
            That’s fine, now we need some volunteer that opens the new threads with some clear description, and hopefully linking to some previous posts/threads regarding that topic (hint: search for posts by insert_nick and shamblett), and a forum moderator that will move the posts in this thread to where they belong.

            These days I can’t do that myself, but it would definitely clean up things because all the topics of the by now called "The Shamblett’s List" are complex enough even alone.

            So shamblett if you feel you can invest I think no less than half an hour to go into this trouble, please do it as I think you are the right one, and let us now in this thread the links to the new created topics so to quickly subscribe them.
              • 34017
              • 898 Posts
              Just wanted to apologize for my interjecting when i assumed the topic was versioning instead of actually reading the posts. Sorry.

              I’ve been working with YAML quite a bit lately and really like it for config lists. It seems you have too nick. Some things gathered from working with Symfony (which used yaml for config files).

              - Loading large YAML files takes a lot of time. Symfony converts them into arrays and saves the file as cached.
              - Even then, checking the timestamps of of files, generating the new DB inserts, then saving that into the db on each request is going to probably be a resource hog and be sloooow.

              Nick, you are saying in the dev environment, compare the fs to the db for each page load and update the db, correct?
                Chuck the Trukk
                ProWebscape.com :: Nashville-WebDesign.com
                - - - - - - - -
                What are TV's? Here's some info below.
                http://modxcms.com/forums/index.php/topic,21081.msg159009.html#msg1590091
                http://modxcms.com/forums/index.php/topic,14957.msg97008.html#msg97008
                • 12983
                • 108 Posts
                Quote from: ChuckTrukk at Oct 20, 2009, 10:27 AM

                Just wanted to apologize for my interjecting when i assumed the topic was versioning instead of actually reading the posts. Sorry.

                Don’t mind, it happens. Now I hope someone will put things clean back, or I fear another year-long stale before some good developer gets focused on this specific topic smiley

                Quote from: ChuckTrukk at Oct 20, 2009, 10:27 AM
                I’ve been working with YAML quite a bit lately and really like it for config lists. It seems you have too nick.

                Yes I do like YAML, especially for the approach I proposed for this thread topic, where I want to keep things clean in files. In my proposal, I see the yaml section like a quick-human-and-machine-readable comment section on the very beginning of each file of the sfs, where you write only properties that you need to alter for that item or just take under your eyes when writing the real file content.

                Quote from: ChuckTrukk at Oct 20, 2009, 10:27 AM

                - Loading large YAML files takes a lot of time. Symfony converts them into arrays and saves the file as cached.
                - Even then, checking the timestamps of of files, generating the new DB inserts, then saving that into the db on each request is going to probably be a resource hog and be sloooow.

                Nick, you are saying in the dev environment, compare the fs to the db for each page load and update the db, correct?

                Yes it’s correct. In hot development time periods, I suggest it for production environment too, or at least turning it "on" in prod when updating, and then back off (but this is left to developer decision, resources available, etc. Again, nothing is imposed, everyone will find his/her way, and it will be probably different form project to project: currently, I have a couple of projects where I know I could afford to stay with it always "on" in the production environment, letting me gain a lot of time I could spend doing other interesting things not related to updating stuff from local to remote).

                The time add estimate should also consider that the only "sure operations" are the check of timestamps of files in sfs and the update date in db fields for documents and elements; these are not the most time consuming operations one could think to, for such a comfort gain - and I suspect that on local environment the difference would be hardly noticeable; then, ONLY the "unsynched twins" will require action, if any; besides, the yaml part on the files (again, just the unsynched ones) won’t be large usually, because generally it won’t contain the whole properties set (for most sfs files, just name and description and some few others will appear in the yaml section).
                  • 12983
                  • 108 Posts
                  Quote from: OpenGeek at Oct 19, 2009, 12:37 AM
                  Logging all the changes made in one deployment and synchronizing them with a production environment via Subversion (for example) will not work in many scenarios, either because during development you tend to "try out" a lot of things that will not go into the end product, and/or because production data is being changed directly in production while development is going on.

                  It’s true the fact that data is being changed directly in production while development is going on; just as an example, I can think to news or blog post being published by the client on a site, or a web designer different from the php coders who independently updates the template whenever he feels to, or several php coders, etc.

                  But concurrent development on files can be managed by traditional version control tools, as long as the sfs-RDM is an option, because it’s all a matter of filesystem stuff.

                  We already develop in team on MODx with a "document-id independent" approach, and with a file-based documents approach, to minimize the efforts in synchronizing our environments, and the development and production environments. But a lot of manual work is still in front of us when we have to move our work from dev to prod.

                  Besides, even if (in Evolution) we use documents to store calls to snippets that load stuff from files as often suggested on this regard, we would like to better use the caching features of MODx, with the "real" content stored into the db: so my sfs-RDM proposal takes this into account, where I want that the filesystem is just "an alternative frontend", but still staying within the rails of MODx, which is designed with real contents copy/pasted into the manager UI fields to be saved in the database.

                  Quote from: OpenGeek at Oct 19, 2009, 12:37 AM
                  I think being able to manage promotion of specific changes though a combination of Registry-based audit logs and Transport Packages, which can contain scripted logic where needed to validate dependencies or resolve conflicts, is going to be the best solution for the widest number of scenarios. And, we can start the work of making this manageable without developer intervention after we identify and work through the specifics of the solution as developers.

                  We already "promote" stuff on production by manually and carefully overwriting the right files in filesystem, and doing the right updates within the manager UI. Of course I agree that a tool allowing for selective promotion of detected changes would be very useful.

                  But this would be an evolution: the core thing, imho is the check on editedon audit fields on db and timestamps of files to see what needs evaluation. If the evaluation is automatic, it’s done by the extension itself and the rule should be to promote the fresh.
                  And yes if we want to look further, also a manual promotion tool could be implemented with this approach, e.g. a "manual" setting in the sfs-RDM extension, with a module in manager where we can browse the pending changes detected and selectively include/exclude those we want from promotion. But I say, better to have the "little thing", and then eventually build on it.

                  Unfortunately I still can’t see how this could be done with the MODx Registry instead of audit fields, so if you have followed my posts and understand what I’m describing, I’d like to support the Registry cause and forget the db/sfs date checks, since it wouldn’t require to wait for 2.1 to have someone good jumping in and starting coding, but really I have a problem in imagining how the Registry thing could be useful for this one (maybe I need a simple example, or better a scheme like the one I wrote in a post above).