We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 22213
    • 52 Posts
    The decision to store all tv values as text is, frankly, awful. I’ve spent two years dealing with aspects of this  bad design decision. The basic incompatibilities between how PHP and Unix think about dates, and how MySQL stores them did not need another layer of confusion where elaborate conversions bound to slow down db operations, complicate code etc. are required.

    This is the sort of place where Modx limits itself. Such a great environment in so many ways, and things like this act as barriers to adoption. I’ve been teetering on the brink of abandoning it for a year because of complications that arise from this.

    I tried to resolve the issue using the method recommended by several in the community -- an elaborate @EVAL in one tv that reads another and writes a value to a third. But this causes grief when updates and duplications happen, which turn out to be important to the uses my client is putting their ModX based site.

    That said, the transfer from one hosting provider to another has provided the occasion to attack some of this cruft. Given the need to have a unixtime string stored whenever the user updates a date tv, instead of the old method, I’ve hacked the save_content.processor.php file. The results are much more satisfactory, and require no db mods, so I’m posting them here in case someone else can benefit from the approach.

    The bit of code I’ve posted below does the following:

    - after any tv updates or new values are written to the db in save_content.processor.php, it scans the _site_tmplvars table for any tvs whose names end in ’_daterow’.

    - it then looks for a tv with the same name, minus the ’_daterow’. So, you might have a tv called ’showStart’, and if you have another tv called ’showStart_daterow’ it’ll find the former.

    - it then looks for a db entry for the current page for the source tv, ie ’showStart’ in my example.

    - it extracts the value, assuming it’s a date formatted as modx stores them with the date widget.

    - it converts this to a unixtime string, and stores it in a ’_daterow’ tv entry for the current page, creating one if one does not exist.

    What all this means is that if you add this to save_content.processor.php, starting at around line 473, (check for code matches at the beginning and end of the code below in your copy),  to store unixtime value semi-automatically, you just need to:

    - create a tv with the date widget for input;

    - create a second, with the same name, and the suffix ’_daterow’.

    Now you can use that second field to sort, search and compare entries.

    Hope that this is useful to someone.

    Needs more error correction, but I assume that this will get reworked anyway....


    
    Deleted
    


    -- removed some inadvertently included test code... DD
      Web Designer
      PHP Programmer
      Cocoa Developer
      Boulevardier & Arriviste
      • 25663 MODX Staff
      • 12,272 Posts
      How does this handle pre-1970 dates?

      Did you ever try using the date-picker input widget and the unixtime output widget? I think you can accomplish the same thing for sorting purposes and use special date-formatting snippets if you need to display the values as human-readable output, only you don’t need to modify anything at all in the parser. I may have read this too quickly though and be missing something.
        Ryan Thrash, MODX Co-Founder
        Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
        • 22213
        • 52 Posts
        The unxitime output widget doesn’t apply to operations that are outside modx’s operations. In fact, a huge problem is that to an external process, there isn’t a way to distinguish data types in tvs, because they’re all generic text. Obviously, in the world we’re in, limiting data to the hermetic world of Modx is non-trivial when I want to mine my content for reuse outside of the Modx context.

        More importantly, the unixtime output widget (AFAIK) is exactly that, an output widget. It doesn’t affect the storage of the data in the db, but simply formats it on the way out. At least not in my tests.

        To reiterate the underlying problem, Modx’s dates (apart from created & published dates) are stored as a formatted string, and because of this, not even mysql can distinguish earlier and later. It’s a terrible, terrible decision and should have never gotten this far into the development. At least a unixtime number stored as a string can still be used for basic operations of before and after and between.

        To take it one step further, Modx lack of an API means that external processes that interact with the data rely on the limits of standard sql, and that can’t deal with dates provided as a string. Given the enormous potential flexibility of the tv idea ( a critical reason I chose Modx for the main project I use it for), it’s regrettable that it throws most of that away by using a private system to determine underlying datatypes.

        The convention of naming a column with a certain extension as I have done could be used to overcome this, but clearly Modx should be imposing that standard if it is going to persist with the everything-is-a-string approach, and not leaving it to individuals to patch the core code to get this effect.
          Web Designer
          PHP Programmer
          Cocoa Developer
          Boulevardier & Arriviste
          • 22213
          • 52 Posts
          Quote from: rthrash at Nov 11, 2008, 05:57 AM

          How does this handle pre-1970 dates?


          What happened before 1970? grin
            Web Designer
            PHP Programmer
            Cocoa Developer
            Boulevardier & Arriviste
            • 30223
            • 1,010 Posts
            Uhm,.. I bought my first microprocessor ? laugh
              • 22303 MODX Staff
              • 10,725 Posts
              Quote from: omnivore at Nov 11, 2008, 08:20 AM

              So, did you go on any dates before you bought your first microprocessor?
              MODx stores native dates/timestamps as integers (though even this is going to change in the future).  The use of TV values as dates is up to you; if you don’t like a particular behavior, ask questions, suggest other behaviors, or otherwise contribute to the project before using verbal attacks that are going to reduce our willingness to respond.
                • 4172
                • 5,888 Posts
                you can search and sort nearly for all cases in date-tvs, see this little sql (some parts of this are from easy-events):

                SELECT DISTINCT 
                sc.id,
                sc.pagetitle,
                sc.description,
                sc.published, 
                sendungstyp_tvcv.value AS sendungstyp,
                moderator_tvcv.value AS moderator, 
                s_tvcv.value AS startDate, 
                e_tvcv.value AS endDate, 
                h_tvcv.value AS hide, 
                IF (sc.parent = 75 ,1,0) AS isdayli, 
                IF (RIGHT(s_tvcv.value,8)<'12:03:43' ,1,0) AS isbefore,
                CONCAT(IF (sc.parent = 75 , 
                IF (RIGHT(s_tvcv.value,8)<'12:03:43' ,ADDDATE(CURDATE(), 1),CURDATE()), CONCAT(SUBSTRING(s_tvcv.value,7,4),'-',SUBSTRING(s_tvcv.value,4,2),'-',SUBSTRING(s_tvcv.value,1,2))),' ',
                RIGHT(s_tvcv.value,8)) AS starttime, 
                CONCAT(IF (sc.parent = 75 , CURDATE(), CONCAT(SUBSTRING(e_tvcv.value,7,4),'-',SUBSTRING(e_tvcv.value,4,2),'-',SUBSTRING(e_tvcv.value,1,2))),' ',RIGHT(e_tvcv.value,8)) AS endtime 
                FROM (`db27029x821689`.`modx_site_content` AS sc, 
                `db27029x821689`.`modx_site_tmplvars` AS sendungstyp_tv,
                `db27029x821689`.`modx_site_tmplvars` AS moderator_tv, 
                `db27029x821689`.`modx_site_templates` AS t, 
                `db27029x821689`.`modx_site_tmplvars` AS s_tv, 
                `db27029x821689`.`modx_site_tmplvars` AS e_tv, 
                `db27029x821689`.`modx_site_tmplvars` AS h_tv) 
                # Start Date 
                JOIN `db27029x821689`.`modx_site_tmplvar_contentvalues` AS s_tvcv 
                ON s_tvcv.contentid = sc.id AND s_tv.id = s_tvcv.tmplvarid 
                LEFT JOIN `db27029x821689`.`modx_site_tmplvar_contentvalues` AS sendungstyp_tvcv 
                ON sendungstyp_tvcv.contentid = sc.id AND sendungstyp_tvcv.tmplvarid = sendungstyp_tv.id 
                LEFT JOIN `db27029x821689`.`modx_site_tmplvar_contentvalues` AS moderator_tvcv 
                ON moderator_tvcv.contentid = sc.id AND moderator_tvcv.tmplvarid = moderator_tv.id 
                # End Date 
                LEFT JOIN `db27029x821689`.`modx_site_tmplvar_contentvalues` AS e_tvcv 
                ON e_tvcv.contentid = sc.id AND e_tvcv.tmplvarid = e_tv.id 
                # Hide time LEFT JOIN `db27029x821689`.`modx_site_tmplvar_contentvalues` AS h_tvcv 
                ON h_tvcv.contentid = sc.id AND h_tvcv.tmplvarid = h_tv.id 
                LEFT JOIN `db27029x821689`.`modx_document_groups` AS dg 
                ON dg.document = sc.id 
                WHERE (sc.privateweb = 0 
                OR dg.document_group IN (1,3)) 
                AND (sc.parent IN (73) OR sc.parent = 75) 
                AND sc.published = 1 
                # Start date 
                AND s_tv.name = 'EasyEvents_Start' 
                AND CONCAT(IF (sc.parent = 75 , IF (RIGHT(s_tvcv.value,8)<'12:03:43' ,ADDDATE(CURDATE(), 1),CURDATE()), CONCAT(SUBSTRING(s_tvcv.value,7,4),'-',SUBSTRING(s_tvcv.value,4,2),'-',SUBSTRING(s_tvcv.value,1,2))),' ',RIGHT(s_tvcv.value,8)) > '2008-10-08 12:03:43' 
                # End date 
                AND e_tv.name = 'EasyEvents_End' 
                # Hide 
                AND h_tv.name = 'EasyEvents_HideTime' 
                AND sendungstyp_tv.name = 'sendungstyp' 
                AND moderator_tv.name = 'moderator' 
                ORDER BY starttime
                  -------------------------------

                  you can buy me a beer, if you like MIGX

                  http://webcmsolutions.de/migx.html

                  Thanks!
                  • 25663 MODX Staff
                  • 12,272 Posts
                  omnivore, it’d be infinitely more productive if rather than ranting in a way that comes across as arrogantly trying to prove intellectual superiority that you’d simply file a bug report in Jira, the appropriate channel for getting things resolved. Rants in the forums aren’t productive and with the volume of posts here issues will get lost or overlooked inevitably. You can find a link to it in the header of the forums under "Bugs & Requests" or in my signature. Any suggestions that work for dates outside of the unix timestamp (pre-1970) would be greatly appreciated.
                    Ryan Thrash, MODX Co-Founder
                    Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me
                    • 22213
                    • 52 Posts
                    @Bruno17


                    you can search and sort nearly for all cases in date-tvs, see this little sql (some parts of this are from easy-events):

                    I appreciate that this might solve the problem. But my observation at the outset was that excessively complex code, operations slowing down the db etc was the problem. This doesn’t seem to avoid that.

                    @Ryan Thrash:

                    omnivore, it’d be infinitely more productive if rather than ranting in a way that comes across as arrogantly trying to prove intellectual superiority that you’d simply file a bug report in Jira, the appropriate channel for getting things resolved.

                    I am acutely aware, as someone who designs and implements web interfaces and backends that it is a painful and unpleasant part of my job to listen to the unvarnished criticism that my clients direct at my efforts.

                    I have, in the past, done what many interface designers do -- lashed out (once the clients left the room) at their stupidity, and at their unpreparedness to do things the way I have, in my supreme brilliance, determined that they should. I’ve observed (to myself) that users who don’t spot the critical button at the right point in the process are, in fact, idiots and unhelpful. And I have decried the arrogance of people who have found flaws in my work without the inclination or ability to do better.

                    And then, I’ve done what anyone committed to a great product does: I’ve calmed down, looked at their contribution and realized that if I drove them to make "arrogant" comments, then I probably had frustrated them unreasonably. I’ve realized that if their committment to and interest in the underlying technologies I use were as great as mine, I wouldn’t be working for them: they’d do it themselves: that my job is to deliver something that makes their life easier, not burden them with new tasks. And I’ve realized that they see my feelings about being criticised as secondary to their committment to using what I have proposed to deliver for the purposes that I claimed it could be put.

                    And I have learned to thank them, sincerely and with the knowledge that I’ve never created anything great without their arrogant, ignorant, unhelpful, lazy and inobservant feedback that shows so little regard for my sensitive creative soul.

                    So far, none of them has made any code contributions to the projects I’ve created for them. I appreciate that their real value is that they’ve taken the time to criticise, however sharply and harshly, and that it’s when you stop hearing from them that you know you’re really screwed. Ultimately they’ve helped me understand that I made the decision to work in what is probably the industry with the highest-rising, fastest changing standards, and if I want to stay part of that industry, I’d do better to greet their contributions that irritate me so much when they deliver them with the greatest gratitude, since without them, I’m just another dreamer, free to imagine that what I do is fabulous, instead of being able to actually make it that way. And they’ve taught me to be generous with my criticism, when I see potential in something, and to be silent with things that don’t matter.

                      Web Designer
                      PHP Programmer
                      Cocoa Developer
                      Boulevardier & Arriviste
                      • 22213
                      • 52 Posts
                      Quote from: OpenGeek at Nov 11, 2008, 08:44 AM

                      Quote from: omnivore at Nov 11, 2008, 08:20 AM

                      So, did you go on any dates before you bought your first microprocessor?
                      MODx stores native dates/timestamps as integers (though even this is going to change in the future). The use of TV values as dates is up to you; if you don’t like a particular behavior, ask questions, suggest other behaviors, or otherwise contribute to the project before using verbal attacks that are going to reduce our willingness to respond.

                      Does providing code that corrects the problem entirely in a couple of dozen lines, easily integrated into any ModX installation and without disruption or change to any other DB or table not count as "otherwise contribute", then?
                        Web Designer
                        PHP Programmer
                        Cocoa Developer
                        Boulevardier & Arriviste