We launched new forums in March 2019—join us there. In a hurry for help with your website? Get Help Now!
    • 23214
    • 36 Posts
    I've noticed an issue with my modx_site_tmplvar_contentvalues table whereby date values appear to get corrupted/deleted.

    This only appears to happen with Date TV's and I suspect it only happens after I edit, delete or remove/expunge documents via the manager.

    If you see the attached pdf file, this contains an export from my modx_site_tmplvar_contentvalues table for tmplvarid = 33. The data on the left is from an automated backup dated 1st May. The data on the right is taken live from the database today (3rd May). You will notice that I have highlighted certain cells in red where the data has been truncated or deleted entirely somehow.

    A large number of changes were made to the content via the manager on the 1st May (after the backup was made). I strongly suspect these content changes (made by me) have caused these issues with this table. The content changes were fairly simple, creating/editing/deleting/expunging documents and the damaged TV content values only appear to be on Date TV's.

    Any thoughts/questions? The site is running on MODX 2.1.3-pl.

    thanks
    Andrew
      • 23214
      • 36 Posts
      Update:

      Some examples (for those of you unable to download the attached pdf):
      Before: "1998-07-01 09:47:00" After: ""
      Before: "2000-10-01 10:05:55" After: "2000-10-01 10:5"
      Before: "2002-10-09 10:15:44" After: "2002-10-09 10:15"
      Before: "1999-12-01 10:38:29" After: ""
      Before: "1997-11-01 10:51:41" After: ""

      Over approximately 80 rows of data there are 20 cases like this, where the data has been corrupted or deleted.
        • 3749
        • 24,544 Posts
        If there a default set for the TV? If the TV is set to the default value, there's no entry in the DB for that TV.

        Also, are you using NewsPublisher or another front-end editing solution? There's a bug in MODX that wipes TV values for TVs that are not included in the edited page.

        That said, it looks like a regular old bug in MODX. You might try turning off compress_js and maybe compress_css in System Settings to see if that makes a difference.


        ------------------------------------------------------------------------------------------
        PLEASE, PLEASE specify the version of MODX you are using.
        MODX info for everyone: http://bobsguides.com/modx.html
          Did I help you? Buy me a beer
          Get my Book: MODX:The Official Guide
          MODX info for everyone: http://bobsguides.com/modx.html
          My MODX Extras
          Bob's Guides is now hosted at A2 MODX Hosting
          • 23214
          • 36 Posts
          Thanks for your reply Bob.

          The default for the TV is blank. I also noticed a similar situation for another Date field (tmplvarid = 54), this also has a default of blank.

          The packages I have installed in Package Management are:

          • babel-2.2.4-pl
          • breadcrumbs-1.1.0-pl
          • formit-2.0.1-pl
          • getpage-1.1.0-pl
          • getresources-1.3.1-pl
          • googlesitemap-1.2.1-pl
          • if-1.1-pl
          • login-1.6.5-pl
          • simplesearch-1.4.0-pl
          • tinymce-4.3.0-rc2
          • ultimateparent-2.0-pl
          • wayfinder-2.3.1-pl

          From that list, I believe only Babel and TinyMCE are used within the manager?

          I should have pointed out that this is a multi-lingual site with two contexts (web, web-fr) for English/French. I know that Babel uses a hidden TV to relate documents belonging to the different contexts. After some testing (below) I'm sure the issue is with Babel.

          I've created a reliable way of demonstrating this bug within a development version of the website which I can test and restore repeatedly. The steps are as follows:

          • Login to manager
          • Delete document #38
          • Expunge document #38

          After doing this, the value field in two rows of modx_site_tmplvar_contentvalues become blank:

          • id: 543, tmplvarid: 33, contentid: 140, value: ""
          • id: 1659, tmplvarid: 33, contentid: 310, value: ""

          Thoughts/comments:

          • This bug only appears when deleting certain documents, I'm not sure why.
          • Each time I repeat the above steps (after restoring the database each time) the same two modx_site_tmplvar_contentvalues always become blank.
          • I've checked the modx_site_tmplvar_contentvalues after each of the above steps, the tv values only go blank after the expunge (i.e. when document 38 is permenantly removed from the database).
          • The TV values which are deleted are related to documents 140 and 310 and have nothing to do with document 38.
          • Documents 140 and 310 do not share the same template as document 38. (Docs 140 + 310 have a template id of 9. Doc 38 has a template id of 6).
          • Template 9 and 6 share the same four TV's: #9, #19, #28, #56 (they do not share TV #33, the one which goes blank).
          • Documents 140 and 310 are related to one another via Babel via the babelLanguageLinks TV (#56) (i.e. doc 310 is the web-fr version of doc 140). This suggests to me that it is a bug with Babel rather than MODX.

          To test this bug I've restored the database and tested the following each time:
          1. Turn off compress_js/compress_css.
          Blank values still appear in modx_site_tmplvar_contentvalues.

          2. Uninstall Babel:
          Bug doesn't occur.

          3. Update Babel to 2.2.5-pl
          Blank values still appear in modx_site_tmplvar_contentvalues.

          4. Update all packages and MODX (to 2.2.2-pl).
          Blank values still appear in modx_site_tmplvar_contentvalues.

          Note: the packages are now:

          • babel-2.2.5-pl
          • breadcrumbs-1.1.0-pl
          • formit-2.0.3-pl
          • getpage-1.2.2-pl
          • getresources-1.4.2-pl
          • googlesitemap-1.3.1-pl
          • if-1.1-pl
          • login-1.7.3-pl
          • simplesearch-1.6.0-pl
          • tinymce-4.3.2-pl
          • ultimateparent-2.0-pl
          • wayfinder-2.3.3-pl

          I will try digging through Babel to see whats happening. For now I'll avoid expunging anything and only expunge after making a DB backup.

          Any other comments/suggestions are welcome.
            • 23214
            • 36 Posts
            Ok, narrowed this right down now and partially fixed it (at least for my installation).

            The following code is taken from Babel's Plugin:
            case 'OnEmptyTrash':
            	/* remove translation links to non-existing resources */
            	$deletedResourceIds =& $modx->event->params['ids'];
            	if(is_array($deletedResourceIds)) {
            		foreach ($deletedResourceIds as $deletedResourceId) {
            			$babel->removeLanguageLinksToResource($deletedResourceId);
            		}
            	}		
            	break;


            When document 38 is expunged this calls: $babel->removeLanguageLinksToResource( 38 );

            removeLanguageLinksToResource() is defined in core/components/babel/model/babel/babel.class.php as:
            /**
             * Removes all translation links to the specified resource.
             * 
             * @param int $resourceId id of resource.
             */
            public function removeLanguageLinksToResource($resourceId) {
            	/* search for resource which contain a ':$resourceId' in their Babel TV */
            	$templateVarResources = $this->modx->getCollection('modTemplateVarResource', array(
            		'value:LIKE' => '%:'.$resourceId.'%'));
            	if(!is_array($templateVarResources)) {
            		return;
            	}
            	foreach($templateVarResources as $templateVarResource) {
            		/* go through each resource and remove the link of the specified resource */
            		$oldValue = $templateVarResource->get('value');
            		$linkedResources = $this->decodeTranslationLinks($oldValue);
            		/* array maps context keys to resource ids
            		 * -> search for the context key of the specified resource id */
            		$contextKey = array_search($resourceId, $linkedResources);
            		unset($linkedResources[$contextKey]);
            		$newValue = $this->encodeTranslationLinks($linkedResources);
            		$templateVarResource->set('value', $newValue);
            		$templateVarResource->save();
            	}
            }


            I've altered this function to:
            /**
             * Removes all translation links to the specified resource.
             * 
             * @param int $resourceId id of resource.
             */
            public function removeLanguageLinksToResource($resourceId) {
            	/* search for resource which contain a ':$resourceId' in their Babel TV */
            	$templateVarResources = $this->modx->getCollection('modTemplateVarResource', array(
            		'value:LIKE' => '%:'.$resourceId.'%'));
            	if(!is_array($templateVarResources)) {
            		return;
            	}
            	foreach($templateVarResources as $templateVarResource) {
            		/* go through each resource and remove the link of the specified resource */
            		$oldValue = $templateVarResource->get('value');
            		$linkedResources = $this->decodeTranslationLinks($oldValue);
            		/* array maps context keys to resource ids
            		 * -> search for the context key of the specified resource id */
            		$contextKey = array_search($resourceId, $linkedResources);
            		
            		/* modification: sanity check, is the contextKey really a context in babel's settings? */
            		if(array_key_exists($contextKey, $this->config['contextKeyToGroup']))
            		{
            			unset($linkedResources[$contextKey]);
            			$newValue = $this->encodeTranslationLinks($linkedResources);
            			$templateVarResource->set('value', $newValue);
            			$templateVarResource->save();
            		}
            	}
            }


            This still isn't a perfect solution though due to the way that the babelLanguageLinks TV is populated.

            Example:

            These are some example TV values for babelLanguageLinks:

            • web-fr:91;web:38
            • web:38;web-fr:91
            • web:138;web-fr:261 (**)
            • web-fr:261;web:138 (**)
            • web:384;web-fr:512 (**)
            • web-fr:512;web:384 (**)

            The getCollection call (on line 8 above) at the beginning of this function effectively searches for all TV values which match "%38%". This would match the first two items above (as expected), but it would also match the remaining four incorrectly (starred **).

            FYI:
            $this->config['contextKeyToGroup'] comes from Babel's system settings. In my installation this is defined as "web,web-fr" (i.e. a comma delimited list of contexts).
            Babel parses this into the following array:
            Array
            (
                [web] => Array
                    (
                        [0] => web
                        [1] => web-fr
                    )
                [web-fr] => Array
                    (
                        [0] => web
                        [1] => web-fr
                    )
            )


            Ideally the removeLanguageLinksToResource() function needs to determine that the id's involved are exactly equal to $resourceId and not LIKE %$resourceId% it.
              • 25506
              • 6 Posts
              Quote from: phatphug at May 04, 2012, 03:15 AM
              The getCollection call (on line 8 above) at the beginning of this function effectively searches for all TV values which match "%38%". This would match the first two items above (as expected), but it would also match the remaining four incorrectly (starred **).

              Yes, the getCollection is a fuzzy match on the resource id, but inside the loop there's a line which ensures that the context key is retrieved for the corresponding resource id:

              $contextKey = array_search($resourceId, $linkedResources);


              This should ensure that partial matches on the resource id won't get accidentally removed.

              However, the getCollection code suffers from another problem, which is that it's checking ALL TVs, and not just the Babel ones. This means that if for some reason a non-Babel TV contained, say, web:38, then that part of the TVs value would be deleted, resulting in data corruption.

              I've modified removeLanguageLinksToResource to limit the getCollection to only the hidden Babel TVs, and for belt-and-braces protection I've added a check in the xPDO to prevent erroneous matches caused by the LIKE (e.g. matching web:384 when looking for web:38):

              /**
               * Removes all translation links to the specified resource.
               *
               * @param int $resourceId id of resource.
               */
              public function removeLanguageLinksToResource($resourceId) {
              	/* search for resource which contain a ':$resourceId' in their Babel TV
              	 * Modified code to ensure that we're only checking the 'babelLanguageLinks' hidden TVs
              	 * The previous code searched all TVs, which risks deletion of TV data that isn't babelLanguageLinks
              	 */
              	$c = $this->modx->newQuery('modTemplateVarResource');
              	$c->innerJoin('modTemplateVar', 'TemplateVar');
              	$c->where(array(
              		'value:LIKE' => '%:'.$resourceId,
              		'OR:value:LIKE' => '%:'.$resourceId.';%'
              	));
              	$c->where(array(
              		'TemplateVar.name' => 'babelLanguageLinks')
              	);
              	$c->prepare();
              
              	$templateVarResources = $this->modx->getCollection('modTemplateVarResource', $c);
              	if(!is_array($templateVarResources)) {
              		return;
              	}
              	foreach($templateVarResources as $templateVarResource) {
              		/* go through each resource and remove the link of the specified resource */
              		$oldValue = $templateVarResource->get('value');
              		$linkedResources = $this->decodeTranslationLinks($oldValue);
              		/* array maps context keys to resource ids
              		 * -> search for the context key of the specified resource id */
              		$contextKey = array_search($resourceId, $linkedResources);
              
              		/* modification: sanity check, is the contextKey really a context in babel's settings? */
              		if(array_key_exists($contextKey, $this->config['contextKeyToGroup']))
              		{
              			unset($linkedResources[$contextKey]);
              			$newValue = $this->encodeTranslationLinks($linkedResources);
              			$templateVarResource->set('value', $newValue);
              			$templateVarResource->save();
              		}
              	}
              }