Snippet to assist upgrade from Evo Ditto to Revo getResources

92 Posts

technicaltitch Reply #1, 10 years, 5 months ago

This is still very much in development. It is working for a few cases, post here early in case anyone knows any gotchas better techniques.

I am upgrading a large, tangled MODx website from Evo to Revo, and it has a lot of very similar Ditto calls, so I created a snippet in Revo called Ditto, which maps Ditto parameters to getResources.

It is the hardest part of my upgrade. If it is useful to anyone else, I figure that as each person adds their use cases to it, it might become actually useful, rather than the messy bit of code you see before you.

In addition to adding this snippet, Ditto dateFormats need to be made explicit in the tpl chunks using the createdon:strtotime:date=`%e %b %y` format, and the paging template needs to be replaced with just [[!+page.nav]]. Other limits of this are noted in code comments.

<?php
global $modx;

$getResourcesParams = array();

if (isset($startID)) 
{
    $parents = $startID;
}
if (isset($parents))
{
    $getResourcesParams['parents'] = $parents;
}

if (isset($sortBy) || isset($sortDir)) {
	$sortDir = isset($sortDir) ? strtoupper($sortDir) : 'DESC';
	$sortBy = isset($sortBy) ? $sortBy : "createdon";
        $getResourcesParams['sortdir'] = $sortDir;
        $getResourcesParams['sortby'] = $sortBy;
}

if (isset($filter))
{
    // only works for exactly two filter conditions OR'd, using filter comparisons 1 (unequal) or 2 (equals), eg, filter=`id,328,2|template,8,1` OR filter=`createdby,2,1|template,5,1`
    // extract the strings as so:    `createdby,2,1|`   =>   $filter_field,$filter_id,$filter_cmp
    $filter_comma1 = strpos($filter, ",", 0);
    $filter_comma2 = strpos($filter, ",", $filter_comma1 + 1);
    $filter_divider = strpos($filter, "|", $filter_comma2 + 1);
    $filter_field = substr($filter, 0, $filter_comma1);
    $filter_id = substr($filter, $filter_comma1, $filter_comma2 - $filter_comma1);
    $filter_cmp = substr($filter, $filter_comma2, $filter_divider - $filter_comma2);

    // Ditto's filter excludes, whereas getResources's where filter includes, so we reverse the boolean operators
    // ie, Ditto &filter=`id,328,2|template,8,1` means exclude where id=328 OR template!=8, equivalent to INCLUDE where id!=328 AND template=8, equivalent to &where=`{"id:!=":328, "AND:template:=":8}`
    // This builds the first half, eg, translates the string   `id,328,2|`   to   `{"id:!=":328, "AND:`
    $filter_cmp_prm = ($filter_cmp == '2') ? '!=' : '=';

    $where = '{"' . $filter_field . ':' . $filter_cmp_prm . '":' . $filter_id . ', "AND:';

    $filter_comma1 = strpos($filter, ",", $filter_divider);
    $filter_comma2 = strpos($filter, ",", $filter_comma1 + 1);
    $filter_field = substr($filter, $filter_divider + 1, $filter_comma1 - $filter_divider);
    $filter_id = substr($filter, $filter_comma1, $filter_comma2 - $filter_comma1);
    $filter_cmp = substr($filter, $filter_comma2);

    $filter_cmp_prm = ($filter_cmp == '2') ? '!=' : '=';

    $where = $where . $filter_field . ':' . $filter_cmp_prm . '":' . $filter_id . '}';
    
    $getResourcesParams['where'] = $where;
}

if (isset($tagData))
{
    $tvFilters = $tagData . '==%' . $tags . '%';
    $getResourcesParams['tvPrefix'] = '';
    $getResourcesParams['tvFilters'] = $tvFilters;
}

if (isset($total))
{
    $getResourcesParams['limit'] = $total;
}
if (isset($depth))
{
    $getResourcesParams['depth'] = $depth;
}
if (isset($tpl))
{
    $getResourcesParams['tpl'] = $tpl;
}
if (isset($tplLast))
{
    $getResourcesParams['tplLast'] = $tplLast;
}
if (isset($hideFolders))
{
    $getResourcesParams['hideContainers'] = $hideFolders;
}

if (isset($paginate)) {
    $getResourcesParams['element'] = 'getResources';
    
    if (isset($display))
    {
        $getResourcesParams['limit'] = $display;
    }

    return $modx->runSnippet('getPage', $getResourcesParams);
}
else
{
    return $modx->runSnippet('getResources',$getResourcesParams);
}

It is all very simple, just time-consuming, and I figure anything to make the Evo->Revo transition easier/cheaper is great for MODx. Thanks to the excellent Provisioner Extra this is the bulk of the effort.
[ed. note: technicaltitch last edited this post 10 years, 5 months ago.]

24,544 Posts

BobRay Reply #2, 10 years, 5 months ago

Very nice. Thanks for the contribution.

I think with a little preg_replace action, it could change the Ditto tags to getResources tags in the DB and avoid the extra overhead of calling runSnippet().

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

92 Posts

technicaltitch Reply #3, 10 years, 5 months ago

Various fixes..

global $modx;

$getResourcesParams = array();

if (isset($startID)) 
{
    $parents = $startID;
}
if (isset($parents))
{
    $getResourcesParams['parents'] = $parents;
}
else
{
    // Ditto defaults to current document's children
    $getResourcesParams['parents'] = $modx->resource->get('id');
}

if (isset($sortBy) || isset($sortDir)) {
    $sortDir = isset($sortDir) ? strtoupper($sortDir) : 'DESC';
    $sortBy = isset($sortBy) ? $sortBy : "createdon";
    $getResourcesParams['sortdir'] = $sortDir;
    $getResourcesParams['sortby'] = $sortBy;
}

if (isset($filter))
{
    // only works for exactly two filter conditions OR'd, using filter comparisons 1 (unequal) or 2 (equals), eg, filter=`id,328,2|template,8,1` OR filter=`createdby,2,1|template,5,1`
    // extract the strings as so:    `createdby,2,1|`   =>   $filter_field,$filter_id,$filter_cmp
    $filter_comma1 = strpos($filter, ",", 0);
    $filter_comma2 = strpos($filter, ",", $filter_comma1 + 1);
    $filter_divider = strpos($filter, "|", $filter_comma2 + 1);
    $filter_field = substr($filter, 0, $filter_comma1);
    $filter_id = substr($filter, $filter_comma1 + 1, $filter_comma2 - $filter_comma1 - 1);
    $filter_cmp = substr($filter, $filter_comma2 + 1, $filter_divider - $filter_comma2 - 1);

    // Ditto's filter excludes, whereas getResources's where filter includes, so we reverse the boolean operators
    // ie, Ditto &filter=`id,328,2|template,8,1` means exclude where id=328 OR template!=8, equivalent to INCLUDE where id!=328 AND template=8, equivalent to &where=`{"id:!=":328, "AND:template:=":8}`
    // This builds the first half, eg, translates the string   `id,328,2|`   to   `{"id:!=":328, "AND:`
    $filter_cmp_prm = ($filter_cmp == '2') ? '!=' : '=';

    $where = '{"' . $filter_field . ':' . $filter_cmp_prm . '":' . $filter_id;

    if ($filter_divider)
    {
        $filter_comma1 = strpos($filter, ",", $filter_divider);
        $filter_comma2 = strpos($filter, ",", $filter_comma1 + 1);
        $filter_field = substr($filter, $filter_divider + 1, $filter_comma1 - $filter_divider - 1);
        $filter_id = substr($filter, $filter_comma1 + 1, $filter_comma2 - $filter_comma1 - 1);
        $filter_cmp = substr($filter, $filter_comma2 + 1);

        $filter_cmp_prm = ($filter_cmp == '2') ? '!=' : '=';

        $where = $where . ', "AND:' . $filter_field . ':' . $filter_cmp_prm . '":' . $filter_id . '}';
    }
    else
    {
        $where = $where . '}';
    }

    $getResourcesParams['where'] = $where;
}

if (isset($tagData))
{
    $tvFilters = $tagData . '==%' . $tags . '%';
    $getResourcesParams['tvPrefix'] = '';
    $getResourcesParams['tvFilters'] = $tvFilters;
}

if (isset($total))
{
    $getResourcesParams['limit'] = $total;
}
if (isset($depth))
{
    if ($depth == 0)
    {
        // Ditto depth==0 means unlimited depth
        $getResourcesParams['depth'] = 200;
    }
    else
    {
        $getResourcesParams['depth'] = $depth;
    }
}
if (isset($tpl))
{
    $getResourcesParams['tpl'] = $tpl;
}
if (isset($tplLast))
{
    $getResourcesParams['tplLast'] = $tplLast;
}
if (isset($hideFolders))
{
    $getResourcesParams['hideContainers'] = $hideFolders;
}

if (isset($paginate)) 
{
    $getResourcesParams['elementClass'] = 'modSnippet';
    $getResourcesParams['element'] = 'getResources';
    $getResourcesParams['pageVarKey'] = 'page';
    $getResourcesParams['includeContent'] = '1';
    $getResourcesParams['includeTVs'] = '1';
    
    if (isset($display))
    {
        $getResourcesParams['limit'] = $display;
    }

    return $modx->runSnippet('getPage', $getResourcesParams);
}
else
{
    return $modx->runSnippet('getResources',$getResourcesParams);
}

92 Posts

technicaltitch Reply #4, 10 years, 5 months ago

Quote from: BobRay at Nov 08, 2013, 11:04 PM

Very nice. Thanks for the contribution.

I think with a little preg_replace action, it could change the Ditto tags to getResources tags in the DB and avoid the extra overhead of calling runSnippet().

Pleasure thanks!

Where do you envisage the preg_replaces being? Do you mean as part of the Provisioner import process? My reservation with that is there are a lot of messy fringe cases so to make this good enough to put into Provisioner would be prohibitive. I've also come across things like bugs in Ditto which the Evo site relied on. I still think this was the quickest approach (especially given I look after 3 other Evo sites for the same client, and one was recently hacked in to) but I think it is unavoidably a code-munging job.

In case anyone reading this is doing a Provisioner upgrade - a valuable lesson in retrospect is to manually import users immediately after you bulk import the rest of the website. If you're handy with SQL you can then update the user ids so that they reconnect as article authors. Unfortunately I created a user so all the IDs were out and I had to update the content createdby ids (as well as manually copying across email address, fullname, etc - Provisioner doesn't even create the user-attributes records). (I'm bitter because a lot of my Ditto calls used User IDs so much of the benefit of this converter was lost...!) [ed. note: technicaltitch last edited this post 10 years, 5 months ago.]

92 Posts

technicaltitch Reply #5, 10 years, 5 months ago

For those estimating a Evo->Revo upgrade, Provisioner maintains document IDs (if you import into a fresh database), so all links still work.

However template IDs are not retained across the import, so any filters etc based on template ID will have to be updated.

24,544 Posts

BobRay Reply #6, 10 years, 5 months ago

Quote from: technicaltitch at Nov 10, 2013, 08:13 PM

Where do you envisage the preg_replaces being? Do you mean as part of the Provisioner import process?

I was thinking of a utility snippet with a bunch of getCollection() calls that would get the content of the objects and make the corrections. It would run after the import, but if you have a lot of oddball Ditto tags, it might not be worth it.

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting

92 Posts

technicaltitch Reply #7, 10 years, 5 months ago

Other bits to look out for are replacing [[+title]] with [[+pagetitle]] in tpls.

For atom feeds, previously [[Ditto? &format=`atom`]], it is much easier to follow the excellent getResources RSS tutorial which gets it up and running in minutes.

Sitemaps just use the excellent GoogleSitemap Revo extra.

Some final fixes on template value processing - I've some templates that compile single pages from several nested resource docs, each with their own template values parsed into the page. Might be some default value fixes in this version too.

I've also left some debug code in there (must admit I had them set to LOG_LEVEL_ERROR as I couldn't get the MODx error level setting working). If you uncomment it, make sure you comment it out again or your database will balloon - it is very verbose.

All now seems to be working for the Ditto calls I have.

<?php
global $modx;

//$modx->log(modX::LOG_LEVEL_DEBUG, 'sortDir  ' . $sortDir);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'sortBy   ' . $sortBy);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'startID  ' . $startID);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'filter   ' . $filter);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'tagData  ' . $tagData);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'tags     ' . $tags);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'display  ' . $display);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'parents  ' . $parents);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'total    ' . $total);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'depth    ' . $depth);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'tpl      ' . $tpl);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'hideFolders  ' . $hideFolders);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'tplLast  ' . $tplLast);
//$modx->log(modX::LOG_LEVEL_DEBUG, 'paginate ' . $paginate);

$getResourcesParams = array();

if (isset($startID)) 
{
    $parents = $startID;
}
if (isset($parents))
{
    $getResourcesParams['parents'] = $parents;
}
else
{
    // Ditto defaults to current document's children
    $getResourcesParams['parents'] = $modx->resource->get('id');
}

if (isset($sortBy) || isset($sortDir)) {
    $sortDir = isset($sortDir) ? strtoupper($sortDir) : 'DESC';
    $sortBy = isset($sortBy) ? $sortBy : "createdon";
    $getResourcesParams['sortdir'] = $sortDir;
    $getResourcesParams['sortby'] = $sortBy;
}

if (isset($filter))
{
    // only works for exactly two filter conditions OR'd, using filter comparisons 1 (unequal) or 2 (equals), eg, filter=`id,328,2|template,8,1` OR filter=`createdby,2,1|template,5,1`
    // extract the strings as so:    `createdby,2,1|`   =>   $filter_field,$filter_id,$filter_cmp
    $filter_comma1 = strpos($filter, ",", 0);
    $filter_comma2 = strpos($filter, ",", $filter_comma1 + 1);
    $filter_divider = strpos($filter, "|", $filter_comma2 + 1);
    $filter_field = substr($filter, 0, $filter_comma1);
    $filter_id = substr($filter, $filter_comma1 + 1, $filter_comma2 - $filter_comma1 - 1);
    $filter_cmp = substr($filter, $filter_comma2 + 1, $filter_divider - $filter_comma2 - 1);

    // Ditto's filter excludes, whereas getResources's where filter includes, so we reverse the boolean operators
    // ie, Ditto &filter=`id,328,2|template,8,1` means exclude where id=328 OR template!=8, equivalent to INCLUDE where id!=328 AND template=8, equivalent to &where=`{"id:!=":328, "AND:template:=":8}`
    // This builds the first half, eg, translates the string   `id,328,2|`   to   `{"id:!=":328
    $filter_cmp_prm = ($filter_cmp == '2') ? '!=' : '=';

    $where = '{"' . $filter_field . ':' . $filter_cmp_prm . '":' . $filter_id;

    if ($filter_divider)
    {
        $filter_comma1 = strpos($filter, ",", $filter_divider);
        $filter_comma2 = strpos($filter, ",", $filter_comma1 + 1);
        $filter_field = substr($filter, $filter_divider + 1, $filter_comma1 - $filter_divider - 1);
        $filter_id = substr($filter, $filter_comma1 + 1, $filter_comma2 - $filter_comma1 - 1);
        $filter_cmp = substr($filter, $filter_comma2 + 1);

        $filter_cmp_prm = ($filter_cmp == '2') ? '!=' : '=';

        $where = $where . ', "AND:' . $filter_field . ':' . $filter_cmp_prm . '":' . $filter_id . '}';
    }
    else
    {
        $where = $where . '}';
    }

    $getResourcesParams['where'] = $where;
}

if (isset($tagData))
{
    $tvFilters = $tagData . '==%' . $tags . '%';
    $getResourcesParams['tvPrefix'] = '';
    $getResourcesParams['tvFilters'] = $tvFilters;
}

if (isset($total))
{
    $getResourcesParams['limit'] = $total;
}
if (isset($depth))
{
    if ($depth == 0)
    {
        // Ditto depth==0 means unlimited depth
        $getResourcesParams['depth'] = 200;
    }
    else
    {
        $getResourcesParams['depth'] = $depth;
    }
}
if (isset($tpl))
{
    $getResourcesParams['tpl'] = $tpl;
}
if (isset($tplLast))
{
    $getResourcesParams['tplLast'] = $tplLast;
}
if (isset($hideFolders))
{
    $getResourcesParams['hideContainers'] = $hideFolders;
}

//foreach ($getResourcesParams as $key => $value)
//{
//    $modx->log(modX::LOG_LEVEL_DEBUG, 'getResourceParams [' . $key .'] = '. $value);
//}

$getResourcesParams['includeContent'] = '1';
$getResourcesParams['includeTVs'] = '1';
$getResourcesParams['prepareTVs'] = '1';
$getResourcesParams['processTVs'] = '1'; 
$getResourcesParams['tvPrefix'] = ''; 

if (isset($paginate)) 
{
    $getResourcesParams['elementClass'] = 'modSnippet';
    $getResourcesParams['element'] = 'getResources';
    $getResourcesParams['pageVarKey'] = 'page';
    $getResourcesParams['pageLimit'] = '10';
    
    if (isset($display))
    {
        $getResourcesParams['limit'] = $display;
    }

    return $modx->runSnippet('getPage', $getResourcesParams);
}
else
{
    return $modx->runSnippet('getResources',$getResourcesParams);
}

92 Posts

technicaltitch Reply #8, 10 years, 5 months ago

Quote from: BobRay at Nov 11, 2013, 04:35 AM

I was thinking of a utility snippet with a bunch of getCollection() calls that would get the content of the objects and make the corrections. It would run after the import, but if you have a lot of oddball Ditto tags, it might not be worth it.

In my case I had to be aware of each call and template anyway, so I prefer the visibility of this approach. And I'm sure bugs remain, both in my shonky code and bugs in Ditto that I haven't catered for in the translation. (getResources seems perfect!)

However inelegant though it is, this definitely saved me time. Being able to fix all bugs in one place, rather than go through each Ditto call, and then discover bugs as I fix subsequent Ditto calls and have to go back, and so on, was nice.

92 Posts

technicaltitch Reply #9, 10 years, 5 months ago

To give an idea of timing, the site I just upgraded had about 1500 pages, about 40 templates, about 70 chunks, 70 snippets, 40 Ditto calls, some doing some pretty complicated stuff, Wayfinder needed a little tweaking, the MODx core code had been mercilessly hacked about by non-developers and I needed to re-implement the functionality outside MODx (I used htaccess), the site was and still is a right mess, with templates called "Duplicate of homepage", files lying around, etc, the (0.9.6) site had been lightly hacked (via the tinyMCE media browser) so all files needed vetting, htaccess was a mess, there were plenty of bugs in the Evo version, MaxiGallery had been pretty heavily customized and needed rewriting in Revo's Gallery (using magnific), blogging had been implemented without Extras, with categories, tags, promotions and all sorts, which needed porting across, templates combined nested documents containing nested and inherited template variables making Ditto calls, into rich sliding Javascript panels, there were 4 or 5 other simple Evo Extras (sitemap etc). Took about 35 hours to upgrade, I think I'm pretty much there.

24,544 Posts

BobRay Reply #10, 10 years, 5 months ago

You might find this useful: http://bobsguides.com/orphans-tutorial.html

It's not 100% reliable, but it will give you a fairly accurate list of unused crap.

You might also want SiteCheck, which will make it really easy to correct any relationships that are invalid, though it's not free: http://bobsguides.com/sitecheck-tutorial.html

Did I help you? Buy me a beer
Get my Book: MODX:The Official Guide
MODX info for everyone: http://bobsguides.com/modx.html
My MODX Extras
Bob's Guides is now hosted at A2 MODX Hosting